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ASSOCIATION TESTS 


I. SCOPE OF THE WORK 


The present paper forms part of. the report of a Committee of 
the American Psychological Association, appointed in 1906, “to 
act as a general control committee on the subject of measure- 
ments.” The Committee consists of Professor James R. Angell, 
chairman, and Professors Judd, Pillsbury, Seashore and Wood- 
worth. This Committee was authorized to organize sub-commit- 
tees and to secure the assistance of other members of the As- 
sociation. A sub-committee on association tests was appointed. 
consisting of the present authors, and the present paper, the 
report of this sub-committee, is to be regarded as a supplement 
to the “Report of the Committee of the American Psychological 
Association on the Standardizing of Procedure in Experimental 
Tests,” published in t910 as No. 53 of the Psychological 
Monographs. 

The Association entrusted to its Committee two general lines 
of work: “first, the determination of a series of group and in- 
dividual tests, with reference to practical application, and second. 
the determination of standard experiments of a more technical 
character.’’ The sub-committee on association tests has confined 
itself to the first of these two lines of work. Leaving aside the 
more elaborate procedure, with chronoscope and lip key, we have 
fixed our attention on the “tests’’ so frequently employed in in- 
dividual and pathological psychology for determining the speed 
and quality of association. Tests are needed which shall not 
require elaborate apparatus nor the expenditure of much time 
on the part of the individual tested. Many such tests are in use; 
these we have attempted to sift and, where possible, improve. 
The manner of giving the tests has varied from one experimenter 
to another ; and we have attempted to ascertain the advantages and 
defects of the different procedures, and to make recommendations 
accordingly. 
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2 R. S. WOODWORTH AND F. L. WELLS 


The efforts of a standardizing committee are likely to be re- 
garded with disfavor and apprehension in many quarters, on the 
ground that the time is not yet ripe for stereotyping either the 
test material or the procedure. It may be felt that what is called 
for, in the present immature condition of individual psychology, 
is, the rather, free invention and the appearance of as many 
variants as possible. Let very many tests be tried, each new 
investigator introducing his own modification; and then, the 
worthless will gradually be eliminated and the fittest will survive. 
Admitting the general justice of this point of view, we still be- 
lieve that work such as is here undertaken may be of service in 
two ways. 

First, we hope that the tests herein recommended may find ap- . 
plication where no special reason exists for the introduction of a 
new test. Often appeals for tests of proved value are heard from 
those who desire to study individual, race, sex, child or patho- 
logical psychology—from investigators who have not the time or 
inclination to devise new tests, and who, moreover, wish to be 
able to compare their results on one class of subjects with results 
already obtained on other classes. If every fresh student employs 
new tests, the incomparability of the results entails much wasted 
effort. Individual and class psychology is, almost of necessity, a 
cooperative enterprise. The advantages to be hoped from stand- 
ardization are much the same here as in the field of an- 
thropometry. 

Second, it can scarcely fail to be true in psychology as in all 
other sciences that a full study of the methods, though too time- 
consuming and too remote from final results to be attractive at 
the start, is certain to lead to more reliable results in the long 
run. In the field of association—aside from the more technical 
experiments in memory—the. methods have not been much sub- 
jected to the kind of experimental criticism which is here at- 
tempted. Usually the investigator has pressed forward to the 
solution of his problem, devising tests that seemed suitable to his 
purpose, and then abiding by them. Our concern being, on the 
contrary, exclusively with the tests themselves, we have sought 
for evidences of their relative value, relying at first on the ex- 
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perience of previous investigators, but in the last resort on re- 
newed experiment with this end in view. 

The tests which we have thus selected are in some degree ana- 
logous to “tested reagents” in chemistry. They make no claim, 
indeed, to be “chemically pure;” that is to say, they can not be 
guaranteed to give a true measure of every individual tested. 
Any mental test is sure to be vitiated in some cases either by 
peculiarities of an individual’s training and information, or by 
the accidental variations to which mental performance is subject 
from moment to moment. These sources of error exist in all 
measurements of intellectual abilities. In the face of such diffi- 
culties, some investigators have felt it necessary to retreat from 
a quantitative attack on individual psychology; while others, 
more hopeful, have sought to neutralize the error of the single 
measurement by statistical methods. In the study of class differ- 
ences, they have relied on averages from large groups; and in 
the study of correlations, they have endeavored to correct for the 
attenuation resulting from chance errors in the single measure- 
ments. But either reliance on the averages of large groups or 
reliance on Spearman’s attenuation formulae is a reliance on 
probability, and therefore sure to be justified in the long run, but 
equally sure to be treacherous somewhere or other. Certainly, 
therefore, it is wise to eliminate from the tests all possible sources 
of error; though other sources of error still remain, yet for every 
defect eliminated there is an increase in the reliability of the in- 
dividual measure, and so of the final result. Now most of the 
tests hitherto employed involve sources of error which can be 
eliminated once they are detected in practise. Many of these 
sources of error are little details in the construction of the tests; 
for example, one or two of the words selected as stimuli may 
have been ambiguous, or unfamiliar to many subjects. Our work 
has very largely consisted in attention to such details; and while 
we cannot hope to have attained perfection of detail, we are sure 
that we have taken some steps in that direction. 

There is general agreement, in practise, as to what shall be 
included under the heading of association tests. There is the 
“free association” test, and the various tests of “controlled as- 


fy 

i 

‘ 

f 
4 
{ 
3 

4 
> 

4 
f 

é 
St 

be 

a 

4. 

7 

a 

> 
i 

4 

ot} 

al 
ike 
f 

od 

4 

+ 
% 
x 

| 

pet 

‘ 


— 


4 R. S. WOODWORTH AND F. L. WELLS 


sociation.”’ In theory, indeed, it is difficult to draw any sharp 
line between association and memory, or intelligence, or reaction 
time. Every mental test involves association; but, in practise, 
the association test is regarded as limited to rather simple in- 
_ tellectual performances, and thus is distinguished from more 
complex tests of intelligence. On the other hand, the association 
between stimulus and response which comes into play in the 
simple reaction, or in the discriminative reaction, is simpler and 
quicker-acting than that of the “associative reaction.’’ The line 
is, however, not sharply defined, and we have included in our list 
of tests one or two (as the “number-checking test’) which might 
be more properly classed under the head of discrimination. 

The distinction between association and memory experiments 
is even harder to draw in theory, though in practise the two are 
well enough distinguished. In the typical memory experiment 
new associations are first formed and later examined as to their 
strength; whereas the association experiment deals with associa- 
tion already formed, and does not control the process of their 
formation. Herein appears an obvious deficiency of the associa- 
tion experiment as compared with the standard experiments on 
memory. The memory experiment deals with a limited system of 
associations, formed specially for the purpose of the experiment 
and under controlled conditions. The association experiment dips 
into the general mass of the individual’s associations, formed at 
various times and under varying conditions, with varying degrees 
of frequency, recency, vividness, emotional and intellectual value ; 
and all these conditions vary from one individual to another. An 
experiment in the formation of entirely new associations gives all 
individuals an equal start; but a test dealing with previously 
formed associations can not hope to be perfectly fair. It aims, 
let us say, to give a measure of the speed of the individual’s as- 
sociative processes; but what it actually measures is, to a large 
extent, the familiarity of the particular associations called for, 
and the freedom of these associations from external inter- 
ferences. 

In the face of these difficulties, the association test may still 
prove of value. It may serve any one of at least three purposes, 


4 
3 
‘ 
j 
. 
A 
f 
H 
4 


SCOPE OF THE WORK 5 


and must be specifically adapted to the purpose which it is re- 
quired to serve. 

(1) A measure of the speed of formation of new associations. 
Such a test is indistinguishable from an experiment in memory 
or practise; but we have included one such, the “substitution 
test.” 

(2) Mental Diagnosis. Here the fact that the same as- 
sociation may have very different values in different individuals 
is fully recognized, and the object in view is to determine the 
value of a given association in the individual. Besides the 
emotional value, of which use is made in “psychoanalysis”, the 
interest of a particular association may be the object of inquiry, 
as in “Tatbestandsdiagnostik.”’ Also, the individual’s familiarity 
with a certain sort of subject matter, or with a given form of 
logical relation, may be the thing measured. Thus the psycho- 
analytic viewpoint in association tests can be used, not only for 
the diagnosis of disturbing ideas and complexes, and for the 
detection of concealed knowledge, but also for showing the lines 
of thought with which an individual is conversant, and the sort 
of relationships along which his mind habitually moves. These 
uses of the association tests often require such close adaptation 
of the experimental material to the special object in view that they 
cannot easily be provided for by a standardized series of tests. 

(3) A measure of mental alertness. The speed of an as- 
sociative reaction depends not only on the strength of the as- 
sociative tendency called into action—and thus on the previous 
training of that association—but also on the “determining tend- 
ency” or “adjustment’”’ or “set of mind.” In controlled associa- 
tion, the speed of the reaction depends on the efficiency of the 
control. In free association, also, a certain adjustment is required 
in order that the stimulus may call out a quick response: there 
must be a receptive attitude, a repression of any train of thought 
that would interfere with the speedy apprehension of the meaning 
of the stimulus; and theré must also be an adjustment to give 
prompt expression to the first idea suggested by the stimulus. In 
a test of either free or controlled associations, calling for a series 
of responses in quick succession to a series of stimuli, the speed 
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6 R. S. WOODWORTH AND F. L. WELLS 


of the performance depends on maintaining the proper adjustment 
throughout the series, in opposition to the many interfering 
tendencies generated by the successive stimuli. Periods of con- 
fusion are apt to occur in the course of such a series; and when 
they occur they impede the action of even well-trained associa- 
tions. One cause of such periods of confusion, as has been 
abundantly pointed out by Jung and his school, lies in the emo- 
tional value of certain stimulus words; but that this is by no 
means the only cause of confusion is made evident in the color 
naming and similar tests, in which the same few stimuli are re- 
peated many times in chance order. The associations required 
are here thoroughly familiar, and usually operate with great 
promptness; but at times they refuse to act properly, so that, in 
the midst of a series of rapid reactions, delayed and even false 
reactions occur to the same stimuli. The confusion here is some- 
times due to wandering of the attention from the work in hand; 
but at other times it seems to be due to interferences generated by 
the performance itself. Whatever may be the cause of confusion 
in each particular case, efficiency in the test requires such a de- 
gree of control as will eliminate the confusion. Periods of con- 
fusion are but extreme manifestations of inefficient control; in a 
minor degree, the inefficiency ‘of one individual in comparison 
with another is shown by uniform slowness of response. 

In order, however, to make the association tests a measure of 
efficient mental control, it is necessary that the associations de- 
manded shall be equally familiar to the individuals compared. In 
strictness, it is impossible to make sure of this; for the experi- 
menter has no sufficient knowledge of the frequency, recency, etc., 
of the training which the several associations have received. The 
best that can be done is to call only for such associations as are 
familiar to all, or at least to the class of individuals to be 
tested. 

Regarded as a measure of mental alertness or efficiency of con- 
trol, the association test should be susceptible of standardiza- 
tion; and the efforts of the sub-committee have accordingly been 
mostly directed to this end. We have in every case but one— 
the Kent-Rosanoff experiment—sought for tests in which the 
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speed of association could properly be taken as the measure of 
efficiency—tests from which the question of the quality of the re- 
sponses could be practically eliminated. To this end we have 
sought to determine, usually by experiment, what associations 
are so generally familiar as to be fair material for a test of 
individual differences in speed of association. We have also 
studied different methods of administering these tests, with a 
view to contributing towards uniformity of procedure; and we 
have, finally, endeavored to furnish average results obtained by 
these tests with one class of subjects, namely young adults of 
fair to good education. 

During the progress of our work, several important contribu- 
tions to the subject have appeared, of which two should be speci- 
ally mentioned, those of Whipple? and of Whitley.?, The aim and 
apparently also the method of Professor Whipple in preparing 
his lists of tests are the same as those of the present report; but 
the scope of his work is much more inclusive, and the present 
paper therefore represents a more intensive study of a limited 
field. The lists of tests here offered may be regarded das sup- 
plementing Whipple’s list at a point where it is not especially full 
nor especially standardized. 

Dr. Whitley’s work is concerned very largely, though not ex- 
clusively, with association and similar tests; and her purpose is 
the same as ours, namely, to test the tests, and determine by 
experiment which are better and which worse. Her methods are 
however different from ours, in that, while we have been princi- 
pally concerned with the details of each test, seeking to eliminate 
defects and sources of error, she has taken a large number of 
tests, as they stood, and compared the results obtained by their 
use. She has tried many similar tests on the same subjects, and 
has moreover repeated the same test a number of times, and then 
has evaluated the tests by the following criteria: (1) the better 
tests should not show rapid improvement with practise, for very 
rapid improvement indicates that some device for dealing with 
the test, or some adaptation to the conditions of the test, is of 


* Manual of Mental and Physical Tests, Baltimore, 1910, pp. 254-270, 312-343. 
*An Empirical Study of Certain Tests for Individual Differences. Archives 
of Psychology, No. 18, 19tt. 
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prime importance; and as some subjects may chance to hit upon 
the adaptation or device at once, and others not, the first trial is 
likely to assign an individual a false position in the function de- 
signed to be measured ; (2) the best test should show only a small 
variation in repeated trials (after the practise effect is allowed 
for), for the greater the variability, the less reliable is the single 
trial or the average of a few trials; (3) the best of a number of 
similar tests is that which correlates most closely with the aver- 
age of them all, for this test represents the fairest sampling of 
the group of similar mental performances which it is desired to 
measure. In point of method, then, Dr. Whitley’s work and 
ours are complementary ; for a good test must both be free from 
minor defects, and must serve to indicate the efficiency of a func- 
tion somewhat broader than that of dealing with the exact ma- 
terial used in the test. In regard to results, it is not easy to 
compare the two pieces of work, so much depends on the particu- 
lar tests examined ; but we find agreement at several points. Dr. 
Whitley, like ourselves, finds the use of written responses inad- 
missible in a test for speed of ‘association; her results also tend 
to give the preference to the use of easy and simple material, such 
as we have adopted; and some of the tests which came out best in 
her comparison—such as an “easy opposites” test, a “first idea” 
test, a letter-checking and a form-checking test, and a form- 
naming test—are véry similar to some included in our list. 
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Il. QUESTIONS OF PROCEDURE 
I. THE FORM OF RESPONSE. 


Where the time of each single reaction is taken, as in the 
classical experiments on association time, the response has almost 
always been a spoken word, and the apparatus has measured 
the time to the beginning of the vocal utterance. But in tests 
which have measured the time, not of each single reaction, but 
of a continuous series of reactions, several forms of response 
have been used. Spoken words, written words, written letters, 
written Arabic numerals, and strokes of the pencil, checking or 
cancelling some of the (visual) stimuli, have all been used in 
different tests. In a test of the speed of any mental process, it is 
clear that the motor expression necessary for experimental pur- 
poses should require as little attention as possible and occupy as 
little time as possible. None of the above mentioned forms of 
response require much attention from an educated subject, but 
speech and cancellation have some advantage in this respect over 
writing. In respect to the time occupied by the movement, also, 
writing is at a disadvantage. The different times occupied by 
these various sorts of motor expression can be judged from the 
following results, obtained from two educated subjects: 


Time for reading (either aloud or silently) a column 

of 20 disconnected letters or Arabic numerals.. 6-7 _ sec. 
Time for reading (either aloud or silently) a column 

of 20 short words, with a total of 22 syllables... 6-71% sec. 


Time for copying 20 one-place numbers........ IO-II sec. 
Time for copying 20 disconnected letters......... 12-13 sec. 
Time for copying 20 short words, containing a total 
Time for cancelling each of a list of 20 letters or 


The oral response, and the cancelling movement, have there- 
fore a great advantage even over the writing of numerals. 


we 
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So slow a process as the writing of words could never be 
thought of as a suitable form of response, were it not for the 
fact, that when a series of stimuli, such as a column of numbers to 
be added or a list of words to which synonyms are required, is 
presented together, the perceptive, associative and motor pro- 
cesses overlap; while the subject is writing the response to the 
first stimulus, he is already dealing with the second stimulus. If 
therefore the motor response is such as to occupy little time in 
comparison with the associative process, the overlapping brings 
it about that the time for the series of responses is nearly identical 
with the time of the associative processes involved; but if the 
motor response takes a much longer time than the associative 
process, the time of the series, because of overlapping, is nearly 
identical with the time of the motor processes. Overlapping 
causes a disappearance of motor time in the first case, and of the 
association time in the second case. So time-consuming a move- 
ment as writing can only be used as an idex of the speed of asso- 
ciation when the associations themselves are much more difficult 
and slow than those which are customary in mental tests. 

With all this admitted, written responses might still find a 
defender, on the ground that the writing should be delayed by 
any halt in the associative process, so that, on the average, the 
longer the time required to write the list of responses, the slower 
must be the association. This is probably true; but it does little 
to weaken the objection to written responses. For, first, if only 
one individual is considered, or only individuals having the 
same speed of writing,—and if, also, the various words to be 
written are suitably adjusted as to length—then the longer 
writing time indicates the slower association, indeed, but 
the indication is far from sensitive, and fails altogether below 
a certain limit. Thus, for example, the associations involved in 
reading a list of words, and those involved in naming colors, are 
both too rapid to be measured by aid of written responses. The 
results of one well-trained subject may be given. To react toa 
series of 20 patches of color by speaking the names required 12 
seconds; to read the 20 printed color-names required but 6 sec- 
onds; but to write the names, either in response to the colors or 
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in response to a list of the names, required in each case 28 sec- 
onds. Here written responses conceal very considerable differ- 
ence in speed of association. Again, in case of the “opposites” 
test, a subject reacted to a list of twenty very familiar stimuli, by 
speaking the opposites, in 15 seconds; to a slightly less familiar 
set in 22 seconds; to a list of the response words, by reading them, 
in 6.5 seconds; but to copy the words from the list required 29 
seconds; to write the responses to the easier set required 31 sec- 
onds, and to the harder set 30 seconds. Thus written responses 
entirely conceal the differences in speed of associations, provided 
only the association time is not over one second; and that even 
without regard to variations in the speed of writing. When 
however different individuals are to be compared, the speed of 
writing must be considered; and as this speed varies at least in 
the ratio of 2 to I, even in educated adults, and as moreover, 
there is no close correlation (as we have found) between the 
speed of writing and the speed of association among educated 
subjects, it is clear, in conclusion, that conditions can scarcely be 
so favorable as to justify the use of written words as responses 
in any test of individual differences in speed of association. 

The case is not quite so unfavorable with the writing of single 
letters or one-place numbers. For example, it is easier to respond 
to a letter by giving the following letter than by giving the 
preceding letter; and this difference appears in either oral or 
written responses. (One subject, 2 trials, list of 20 letters: 
Preceding letter: oral, 32 sec. ; written, 35 sec. Following letter: 
oral, 20 sec.; written, 25 sec.) The writing of single letters or 
numerals is an admissible form of response when the association 


time is over a second—provided the individuals tested are ac- 
customed to rapid writing. 


2. MEASUREMENT OF ASSOCIATION TIME. 


As already remarked, the purpose for which the present set of 
tests is designed excludes the use of elaborate apparatus and 
therefore of the chronoscope and lip key. The custom of many 
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students of association time, in clinical and similar work, is to 
employ the stop watch, starting the watch together with the 
spoken stimulus word, and stopping it on hearing the beginning 
of the response. This procedure would seem to include the ex- 
perimenter’s simple reaction time(probably 150-200 o under the 
conditions) in the measured time. Moreover there is no guaran- 
tee that the watch is started precisely together with the giving of 
the stimulus; a degree of error must be expected here; and more- 
over, the fifth of a second of the stop watch is scarcely to be called 
a fine unit. In spite of these objections, the use of the stop watch 
appears to be justified in practise, especially since the variation in 
association time is so great that significant differences can usually 
be established even with a rough procedure. 

Another procedure, much employed when the chronoscope can 
not be used, is to expose (visually) a whole list of stimuli, and 
to require the subject to react to these in succession and without 
delay between the separate reactions. The time is then taken, not 
for the single reactions, but for the whole series. As the time 
necessary for reacting to the whole series is usually at least Io 
seconds, and often much greater, the deficiencies of the stop 
watch are not serious in this procedure. As indicated in the pre- 
ceding section, when the motor reaction requires little time or 
attention, the overlapping of motor and central processes brings 
it about that the time of such a series of responses is essentially 
central time. If therefore the total time of the series of reactions 
be divided by the number of responses in the series, the quotient 
should give the average association time. 

It would seem possible, indeed, that overlapping should accom- 
plish more than this, and make the average association time, com- 
puted as just described, considerably less than that obtained with 
single stimuli. Cattell’ found that a series of disconnected words 
could be read at a rate of 200g per word, whereas the reading 
time for an isolated word was 360. But in even slightly more 
difficult reactions, such as naming presented colors, this shorten- 
ing of the reaction time, when a series of stimuli is presented 
together, does not appear; but the average time comes out at 


*Wundt’s Philos. Studien, 1885, 2, 635. 
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from 600 to 1200¢. The same holds good for such associations 
as are involved in the opposites test. We have tested five individ- 
uals with the same stimulus words, first singly, and then, several 
months later, in lists. Though the first of these tests should 
have made the responses somewhat more familiar, only one of 
the five subjects reacted more quickly to the words in lists than 
separately ; two subjects reacted more slowly to the words in lists, 
and two showed no marked or consistent difference. On the 
whole, the average time as obtained by timing lists of associative 
responses is no less, and probably somewhat greater, than that ob- 
tained from separate reactions. 

Some explanation is demanded by the failure of overlapping 
to hasten the reaction to a series of stimuli. The explanation is 
probably found in interferences generated in the course of a 
rapid series of associations. Many associative tendencies are 
partially aroused by each stimulus word, and when no interval 
elapses between the successive reactions, the tendencies generated 
by the earlier members of the series must be held in check in 
order to give free play to the associations required by the later 
stimuli. Irrelevant associations enter and tend to impede the 
progress of the reactions. Introspection makes this view seem 
probable, for often the subject is conscious that trains of thought, 
started by the earlier stimuli, must be repressed in order to do 
justice to the later stimuli. Sometimes the response made to a 
stimulus is not wholly satisfactory to the subject; sometimes a 
second response to the same stimulus is suggested immediately 
after the first has been spoken; sometimes an interesting idea or 
disturbing emotion is suggested by a stimulus or by the response 
made to a stimulus. All such interferences die away with the 
lapse of a few seconds between the stimuli; but are present 
in full force when no interval is allowed. Success in dealing 
rapidly with a series of unrelated stimuli requires a higher degree 
of control than success in dealing with isolated stimull. 

This interpretation of the list or serial test is borne out by the 
following experiment.’ The subject had before him a list of 

* We are indebted to Mr. Franklin B. Pedrick for collaboration in this ex- 


periment. It is intended to present elsewhere a fuller report bearing on 
the question of fatigue within brief periods of mental work. 
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20 stimulus words, to which he reacted in quickest possible suc- 
cession ; but the experimenter, instead of timing simply the whole 
list, took the time for each reaction, or, at least, the interval be- 
tween each two successive reactions. This was accomplished by 
bringing side by side on a revolving drum a Jacquet chronograph 
marking fifths of seconds and an electromagnetic marker connect- 
ed with a telegraph key on which rested the experimenter’s finger. 
The experimenter pressed the key on exposing the list to view, and 
then on hearing the beginning of each successive response of the 
subject. Thus a record of the distribution of time through the 
series of responses was obtained, having an accuracy somewhat 
superior to that obtained ordinarily with the stop watch. The 
method can not be employed where the series of responses is very 
rapid and regular (as in naming colors), for then a rhythmic 
tendency dominates the experimenter’s hand; but when the in- 
tervals between responses vary irregularly from 0.4 to 2 or more 
seconds, the method is perfectly feasible. Nine subjects were so 
tested, each reacting to 20 lists of 20 words. The instructions 
called for supraordinate, subordinate concepts, etc., the task re- 
maining the same through each list of 20 stimulus words. (The 
experiment was at the same time designed to indicate the com- 
parative difficulty of the stimulus words, and so to aid in selection 
of the best lists.) 

In combining the: results obtained from several lists and from 
several individuals, with the object of determining the general 
distribution of time throughout the list, difficulty arises from the 
inequal difficulty of the lists and from the unequal speed of the 
individuals. If the times for all the first reactions are simply 
averaged, and so for all the second reactions, etc., the general 
tendency is obscured by the extraneous variations so introduced. 
We therefore proceeded as follows: Taking one individual’s 
performance in response to one list, we determined the distribu- 
tion of time throughout this one list, by first determining the 
average time of these 20 reactions, and the average deviation of 
the reactions, and then expressing the time of each reaction as + 
or — (according as the time of this reaction was greater or less 
than the average time of the twenty) such and such a per cent 
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QUESTIONS OF PROCEDURE 15 


of the average deviation of the reactions in that list. For 
example, a mark of —50 meant that the time of a reaction was 50 
per cent of the average deviation less than the average time for 
the list. The same process was repeated with each of the 20 
lists; and the marks so obtained were averaged for each position 
within a list. Thus an average of + 50 for an individual in the 
first place meant that his first reaction occupied, on the average, 
50 per cent of his average deviation more than his average time. 
The same process was repeated for each individual, and the in- 
dividual marks were averaged. This procedure, then, eliminates 
the absolute times, and also the absolute variabilities, and gives 
an average picture of the relative distribution of time throughout 
the list of twenty. The net result, on the average of the nine 
subjects, is as follows: 


+35 —50 —38 —22 —7 —I7 —25 —32 —16 —I0 
+14 +26 +2+34 +20 +32 +18 +6-+22 —4 


If the speed of reaction were uniform through the list of twen- 
ty, the average mark should be close to 0 throughout; but this is 
not the fact. In spite of the considerable variations and the 
rather large P.E., there can be no doubt that reactions 2-9 tend 
to be quicker than the average, and reactions 11-19 slower than 
the average. There is a slight slackening in the speed of reaction 
throughout the list. The first and last reactions are exceptions to 
this rule; for the first is slow, and the last is more rapid than 
those which immediately precede it. In regard to the first, indi- 
vidual differences are here very great and: characteristic; and a 
fairly strong negative correlation (Pearson r == — 0.69) appears 
between the time of an individual for the whole list and his rela- 
tive time for the first word. This correlation is seen in the accom- 
panying table. 
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Relative time for first 


reaction 
Individual Time for list of 20 in per cent. of av. deviation. 
29.0 +103 
35.6 + 35 


The individual who is relatively slowest in the first reaction 
reacts more rapidly to stimuli in series than to single stimuli, and 
in this respect is rather exceptional. Probably he manages the 
“overlapping” of the successive acts better than most individuals. 
Accordingly his relative slowness in the first reaction may be 
probably explained as due to the necessary absence of overlapping 
at the start. 

The fact, however, that the reaction to the first word of a list 
1s on the average slower than the reaction to an tsolated word 
shows that something besides overlapping and its absence are in 
question. Sometimes a subject, reported that, in glancing at the 
beginning of a list, his eye had caught the second word along with 
the first, and that he was busied with the reaction to the second 
as early as with that to the first. It even happened, occasionally, 
that the reaction to the second word was ready before that to the 
first. This form of interference, incidental as it is to overlapping, 
would of course slacken the reaction to the first stimulus. 

Two influences operate in reacting to a list that are absent in 
reacting to a single stimulus: interference and overlapping. The 
latter tends to accelerate the reactions, the former to slacken them, 
as compared with a reaction to an isolated simulus. Overlapping 
can not exert its accelerating effect upon the first reaction; and 
interference, also, would usually not become operative at the start, 
but special conditions, such as seeing the second word simultan- 
eously with the first, may cause interference to be strongly evi- 
denced at the very start. 
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Aside from the slowness and great variability of the first re- 
action, the most salient fact resulting from the above experiment 
is the quickness of reactions 2-10 as compared with reactions II- 
19. Why should the speed decrease from the second reaction till 
near the close, and then increase again? “Fatigue” and “end- 
spurt” are the catch-words that readily occur to mind; but neither 
of them is specially explanatory. As for fatigue, so short a per- 
formance can hardly cause much fatigue of the genuine, metabol- 
ic sort. Interference seems to be a more probable conception. 
Each succeeding stimulus, and each reaction, tend to evoke asso- 
ciations that are of no service for the purpose of the test. These 
must be repressed ; all their allurements brushed aside. A straight 
course must be steered in spite of many cross currents. As these 
deflecting tendencies continually accumulate with the addition of 
fresh stimuli and reactions, the likelihood of disturbance in- 
creases. 

The increase in speed at the very close can probably be under- 
stood as incidental to overlapping; for, though overlapping leads 
on the whole to increase in speed, it does require, at every moment, 
a division of activity between two or more reactions. At the 
close, this division of activity ceases, and the last reaction receives 
the benefit of the overlapping without any of the incidental draw- 
backs such as were mentioned above in relation to the initial re- 
action. 

The interest of the above experiment, in connection with the 
matter of tests, is the demonstration that the list test brings in 
factors—call them interference and overlapping, or call them 
fatigue, end-spurt, etc—which are not present in reactions to 
isolated stimulus words. The list test reaches a more complicated 
mental performance and calls for a higher degree of control. 

It was desired to see whether a shorter list would show the 
same time-curve as the list of twenty, and whether a list of ten 
words might not be essentially equivalent to ten separate 
stimuli. The experiment was of the same general character as 
above described, but was done in a rougher way. Instead of 
employing a rotating drum, the experimenter held the watch to 
his ear and with his pencil made wavy lines in time with the tick- 
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ing of the watch—the record so left resembling the trace of a 
tuning fork. At each reaction of the subject, the experimenter 
made a break in his time curve, and thus recorded the time of 
each single reaction. The accuracy of these times is about equal 
to that of ordinary stop watch readings. Meanwhile, by consult- 
ing his watch face at the end of the list, the experimenter had the 
time for the whole list. This double use of the watch can be 
recommended when list-tests are used, for the record of the single 
reaction, even if not highly accurate, is of value as showing how 
much of the time is lost in a few slow reactions, and as making 
possible the calculation of the median as well as the average time. 
Some practise is of course necessary before the experimenter can 
successfully use this device. 

-The results of this experiment were treated by the same statis- 
tical method as above described for the preceding experiment. 
Thirteen subjects served, each reacting to 13-24 lists, the total 
number of lists being 243. The average distribution of time in 
a list is shown in the following table, the explanation of which 
is the same as given on p. 15 for the preceding table. 


Position in list ........ I 6 7 8 9 10 
+179 —45 —33 —25 —II —2—I19—I0 +3 —4I 


v 


It is quite possible that the very long relative time indicated for 
the first reaction is in part an artefact; but there is no doubt 
that the first reaction is slow, and that the last reaction is quicker 
than those that immediately precede it, just as was the case in 
the list of twenty. Further, there is a gradual increase of reac- 
tion time from the second to the ninth reaction. This slackening 
is less marked than in the list of twenty, but it is still present to a 
degree. It appears in the average results of 9 out of the 13 indi- 
viduals ; and the 4 who do not show a slackening show no progres- 
sive change in either direction. The conclusion is that the same 
factors are operative in the shorter as in the longer list, though 
not to as high a degree. 

The results of the preceding experiments may well be compared 
with those of an experiment in which the stimulus words were 
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presented orally and separately. Five subjects were examined, 
each 16 times, with the same set of 20 words in different orders. 
The time-curve for these results presents an entirely different 
picture from that obtained in the preceding experiments. The 
first reaction is not slow, but on the average, one of the quickest; 
the last reaction is, on the average, one of the slowest, and yet 
there is no progressive slackening of the reactions from the first 
to the last, but the speed remains, on the whole, very uniform 
throughout. The difference in the time curve of the two modes 
of procedure is probably to be explained by reference to inter- 
ferences : when the series of reactions is continuous, interferences 
tend to accumulate with the progress of the series, but when a 
brief interval of rest intervenes between the successive reactions 
the interferences tend to disappear. It may be concluded that the 
continuous reaction to a series of stimuli is a more complex pro- 
cess than the reaction to a single stimulus, and requires a higher 
grade of control. The two forms of test are not therefore equiva- 
lent, and each may be a good test; but, for a start, preference 
should be given to the simpler form, namely to the reaction to 
separate stimuli. Some associations, however, such as the naming 
of colors or other familiar objects, or the simplest arithmetical 
associations, are too rapid to be timed, singly, by the stop watch. 

There is another advantage in the timing of single associative 
reactions over the timing of a series. The latter method gives 
indeed the average association time for the stimuli used, but 
(unless supplemented by some such device as employed above for 
getting the single times) it shows nothing of the distribution of 
the association times. In particular, since a series is likely to 
contain a few reactions much longer than the rest, the average 
time is apt to differ considerably from the median or the mode, 
and therefore not to be fully typical. The very slow reactions 
are usually due to rather special causes, and their great influence 
on the average is undesirable. The best procedure would seem to 
be that of timing the single reactions, and using the median as 
the typical measure. 

Where a list-test, or continuous test, is employed, our experi- 


ence leads us to favor a rather short list. A list of ten stimulus © 
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words can be timed with sufficient accuracy, and it is freer, on 
the whole, from interferences of a disturbing character. In spite 
of this judgment in favor of shorter lists, we have presented 
lists of considerable length, partly to provide for use with a time 
limit—which is the procedure favored by Professor Thorndike, 
Dr. Whitley, and some other investigators of wide experience— 
and partly to provide a sufficient list of stimuli for separate reac- 
tions, when that is the method adopted. When the continuous 
method is used with an amount limit, it would be better to cut 
the lists in half and take two readings. Two short tests are 
better than one long one, because the average of the two is freer 
from the influence of momentary disturbances, and because it 
allows better for the effects of adaptation to the novel conditions 
of the test. 


3. INSTRUCTIONS TO THE SUBJECT. 


The necessity of uniform instructions has often been insisted 
upon, and sometimes the instructions have been reduced to a set 
formula, in order that all individuals tested, receiving the same 
instructions, may be treated alike. A set formula is, however, 
no guarantee that the subjects are treated alike, for some may 
not comprehend the formula. With a rigid form of instructions, 
the test becomes partly one of the individual’s ability to under- 
stand the instructions, and only partly a test of the function exer- 
cised by the test material. It would be better to provide separate 
tests for ability to understand instructions, and eliminate this fac- 
tor from other tests, so as to make each, as far as possible, a 
test of one function. Proper comprehension of the experiment 
by the subject must not be sacrificed to an ideal uniformity of 
instruction. It matters little by what method is attained the uni- 
form result of the understanding of the experimental task. How- 
ever, there can be little question that the best method for this 
result is that of “learning by doing,” and that the subject should 
learn and demonstrate his capacity for the prescribed reactions 
by going through them. Instruction should proceed by descrip- 
tion, illustration, and execution. The subject should first be 
told clearly the nature of the test; then if possible he should see 
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the operator perform some example of it, and finally he should 
execute samples of new material himself. None of the prelimi- 
nary samples should duplicate the actual test material; nor should 
the preliminary trials be multiplied beyond what is necessary to 
insure the understanding of the test and the first strain of adapta- 
tion to it. Asa rule it may be wise to allow the subject to make 
correct reactions to two samples before passing to the actual test. 

Samples for use in instructing the subject should be prepared 
beforehand and lie ready to the experimenter’s hand. For his 
convenience, it has seemed best to us to provide blanks containing 
samples of most of the tests—one blank containing samples for 
several tests. 

The number-checking tests and the directions test—to be de- 
scribed later—require not oral but written responses, and their ex- 
ecution uses up the blank provided for each subject, and also the 
sample used by each subject. Since this is not the case with the re- 
maining experiments, the instructional material of these two tests 
is best kept separate from that for the remainder. It is repro- 
duced’ on the accompanying page (“Instructional Material I’). 
For the two forms of the number-checking test, two lines of ma- 
terial, organized in the same way as for the actual test blank, are 
provided. For the directions test, there are three directions sim- 
ilar to those on its actual test blank. For the remainder of the 
tests, the same blank may be used for instructing an indefinite 
number of subjects (“Instructional Material II”). For each of 
the forms of addition test, successions of seven figures are sup- 
plied, the subject after verbal instruction and illustration reacting 
to these precisely as to the subsequent test material. There are 
three sample words for the opposites test, and also three sam- 
ples each for the considerable number of tests of partially con- 
trolled association. The substitution test and the color naming 
test supply their own instructional material. The six words 
provided for the free association test are contained neither in the 
Kent-Rosanoff experiment nor in the supplementary thousand- 
word list. 


*Here, and in some other cases later on, the exact type, etc., of the blank 
is not reproduced. 
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INSTRUCTIONAL MATERIAL I, 


Number Checking Test. 


Form A. 45879236017418605923506084231782130756494582763001 
76084395 


Form B. 215864 381592 8267390 967814 371245 942861 
876305 200517 712083 308459 326748 258647 


Directions test: 


1. Write any number larger than 16. 
2. Add one more dot to the largest group 


3. Put a cross over the angle that opens iencinaei Vv 


INSTRUCTIONAL MATERIAL II, 

Addition Test. 

Kraepelin Form Constant Increment Form Opposites Test 
4 32 better 
9 47 
3 21 glad 
8 53 
6 39 straight 
5 28 
2 65 

Logical Relation Tests. 

vb-obj supraord subord pt-wh 

cut horse flower roof 

buy Paris lake tail 

bend potato game Germany 

wh-pt agt-act act-agt att-subst 

wheel train shines cold 

Europe frog howls cheap 

brush sun crawls - narrow 


Mixed Relations Test. 
Box—square 


Orange— 


Woman—husband Man— 


East—west 
Penny—copper 
Asia—China 
Grain—sand 
Am—was 


Free Association Test. 
fox 


apple 
fork 


Day— 
Nail— 
Europe— 


Drop— 
Have— 


cure 
quick 
grass 
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Even after thorough and apparently successful instructions, it 
will occasionally happen that the subject’s reaction to the actual 
test material shows at once that he is on the wrong track. In 
such cases the test must be called off. But if the experimenter is 
provided with pairs of equivalent tests—as can always be accom- 
plished in the present series, either from the provision of dupli- 
cate test blanks, or from the cutting of a long blank into two, as 
previously (p. 20) recommended—then the material of the first 
blank may be used for further instructional samples, and the ac- 
tual test carried out with the equivalent blank. 

There is some difficulty in bringing all subjects to an equality 
in their attitude towards the matter of speed. Occasionally it 
happens that an individual does not try for speed, but only for 
accuracy or distinction or even for an introspective study of his 
performance. Since the time of the performance is the impor- 
tant matter in utilizing the results, it is unfortunate when a good 
subject fails to make an effort for speed. We have seen the stand- 
ing of an individual among his fellows completely changed, in 
the middle of a series of tests, by his being informed that his per- 
formance was slower than the average; his times were at once 
cut nearly in half, while his accuracy was not lessened. We re- 
commend that the instructions include some such statement as the 
following: “The main thing that we are after is to see how rap- 
idly your mind can act. You need not be afraid to put on speed, 
for the test is easy and you are not likely to make mistakes. Of 
course, you should keep on the right track and not make mis- 
takes, and for every mistake you will be docked a little—about 
two per cent” (or one per cent if the whole of long blanks is 
used). The docking by two per cent is, of course, purely arbi- 
trary ; and it may be desirable with some classes of subjects to 
make a larger correction for errors; but our experience with these 
simple tests has not revealed the need of any corrections at all for 
errors. It is probably wise, however, to mention the possibility 
of errors at the beginnmg, and to have an understanding, at 
least with mature subjects, as to the degree of importance attach- 
ed to them. 
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CANCELLATION TESTS—THE NUMBER-CHECK- 
ING TEST 


Controlled association tests are, in method, somewhat analo- 
gous to choice reaction experiments. Like them, they may in- 
volve a certain response to every stimulus, according to a prear- 
ranged scheme of reaction (the B-method of Wundt), or they 
may involve one single reaction to a single kind of stimulus 
among a heterogeneous group of stimuli, the C-method of Wundt. 
Here the subject either reacts or does not, 1. e. has the choice be- 
tween movement and rest. The present test is the only represent- 
ative of this method among the experiments to be described. The 
general idea is the presentation to the subject of a blank upon 
which are printed a large number of different letters, figures or 
designs, and requiring the recognition of each of a certain symbol 
to be indicated by marking. There are measured the speed and 
accuracy with which this is done. This experimental conception 
is not a novel one, having had its origin something over 15 years 
ago, and having, in various forms, played a part in a considerable 
number of subsequent investigations. In company with the 
diverse forms of material that are available for its perfor- 
mance, it has also borne a diversity of names; the most familiar 
single form is probably the so-called A-test, first mentioned by 
Cattell and Farrand, which has long played a part in the Columbia 
Freshman Tests as a measure of “rate of perception.” As a 
rule, it seems desirable to know a test rather by an individually 
descriptive title than by an at best somewhat vaguely defined men- 
tal function with which it may be related. We thus offer the 
present attempt at the standardization of this method under the 
name of the ‘““Number-checking Test; prepared for special pur- 
poses in two forms. 

Form A. As this is, so far as we know, the first time that a 
form made up exclusively of numerals has been offered for this 
purpose, it may not be amiss to give some account of the consid- 
erations which led to their adoption. The use of ordinary con- 
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textual material has its advantage in certain individual applica- 
tions of the test, but is out of place in standard form, because the 
content of the text employed disturbs the attention of the sub- 
ject to the experimental task, and disturbs it in different ways and 
in different degrees for different subjects, thereby violating the 
first principles upon which these tests are constructed. All this 
quite apart from its absolutely chaotic arrangement of the signi- 
ficant stimuli. Pied type obviates most of these difficulties, es- 
pecially if methodically arranged; yet even here it is difficult to 
avoid vocable or other combinations of character that would have 
significance external to the test.’ Geometrical forms satisfy the 
conditions better, but here arises the difficulty of finding a proper 
number of distinct geometrical forms, small enough for the re- 
quisite purpose, and recognizable with sufficient readiness; for 
every effort must be made to obviate false reactions. These con- 
siderations seemed to point pretty definitely toward the use of 
Arabic numerals. They are as readily recognizable as the letters, 
and the chance of any specially suggestive collocation is in- 
finitesimal in comparison with the letters.. The number of the 
symbols, 10, lends itself logically to the use of 100 of each symbol 
in a blank of 1000 symbols, which is as long as there is any 
necessity of making such a blank. 

The general character of the blank being thus determined, 
the arrangement of the material took place as follows: Twenty 
lines of fifty symbols each, properly spaced and justified, provided 
when printed in the regular eight point type a printed space 
with suitable margins on the regular blank sheets of the experi- 
ments. The arrangement of each line is such that it contains five 
each of the symbols 1,2,3,4,5,6,7,8,9,0. Each successive fifth of 
the line (10 symbols) contains each of the ten different symbols 
once. This arrangement of itself obviates the occurrence of 
“runs” of two or more of the same symbol, except at the begin- 
ning and end of each ten, and it was not allowed to occur here. 
The first ten lines of the. blank being completed in this way, the 
second ten lines were constructed by reversing this arrangement. 
This procedure assured the approximate equality of the two halves 


of the experiment as well as the uniformity in the distribution of 
all characters throughout. 
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Special researches may call for different blanks prepared on 
the same principle as those submitted. A great number of blanks 
may be derived from the present one by methodically replacing 
each figure by some other, thus everywhere substituting 1 for 0, 
2 for 1, etc. The test has many desirable features as a measure 
of fluctuations in continued work. Where it is used for this 
purpose a much longer blank would probably be desirable, in 
which the subject should check a greater variety of numbers, 
say all the odd numbers. The time for every line could be noted, 
or if greater precision were required, the time could be reported 
electrically by the graphic method.1./ The experiment should 
also be especially convenient in studies of interference, requiring 
a subject practised for one number to check different numbers, 

and the like. 

’ With the above arrangement, similar as it is for all symbols, it 
makes no difference, so far as the arrangement is concerned, 
which one of the ten symbols the subject is instructed to check. 
The symbols do not however appear to be equally easy and their 
order in this respect should follow their order of distinctiveness ; 
so far as is determined, 1 and 7 seem to be the easiest, next o 
and 4, next 2, 3, 5, 8, and the hardest 6, 9. In the present ex- 
periments the symbol o has uniformly been the one checked. 
Since some subjects may discover essential features in the ar- 
rangement of the blank while others do not, and also with a view 
to obviating omissions, the subject should be uniformly instructed 
that there are in each line five of the symbol they are to check. 
This reduces omissions to an insignificant minimum, if it does not 
obviate them altogether; and this advantage probably more than 
compensates for the occasional delays that result from missing 
the next symbol in succession and having to go back and search 
for it. In practise the procedure recommended seems the more 
desirable with the subjects so far employed. 

After the subject has shown proper understanding of the ex- 
periment by correctly executing the two similarly arranged lines 
of the instructional material, he begins the execution of the test 


*Cf. von Voss, Ueber die Schwankungen der geistigen Arbeitsleistung, 
Ps. Arb., Il, p. 300. 
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blank according to the directions already outlined. A pencil 
should be used for checking, and while it were easy to appear 
hypercritical in this respect, even in so minor detail as the charac- 
ter of the pencil, approximate uniformity should be sought for 
among all subjects between whom direct comparison is to be 
made. It should be one of moderate softness, not harder than 
the No. 2 grade, otherwise the markings may require more than 
the accustomed writing effort to make them properly distinct. 
The timing of the complete execution of-the blank is by the stop- 
watch, and it is advisable to take the time for the halves also; un- 
less the watch is provided with a split-stop, it is scarcely feasible 
to take the half-time closer than the nearest second. 

Probably no other of the present tests shows such irrelevant 
differences in the manner in which different subjects execute it. 
The check-marks vary from light, almost haphazard strokes, to 
heavy, labored scorings. It is impossible, even were it desirable, 
to secure uniformity in these respects. In such matters the sub- 


ject must adopt his own optimum method. While it is not clear 


that any special advantage accrues from this procedure, the same 
must apply to the habit of some subjects to check alternate lines 
backward. More significant data are probably obtained by merely 
noting whether the subject modifies the more natural behavior to 
the alternating directions, and if this is done immediately. A 
few subjects, having checked the two lines of the instructions 
blank, have shown a tendency to check only the first two lines of 
the experimental blank, and subjects should clearly understand 
that the entire blank is to be gone through. Half of the blank is 
however long enough. 

On rare occasions a wrong symbol is checked ; this seems to be 
more frequent in the later stages of practise. As with the omis- 
sions, the influence of such errors is due less to their direct dis- 
tortion of the final time than to the fact that the subject is quite 
apt to observe them and be disturbed by their occurrence. On the 
basis of false reactions, the test has not shown, in the writers’ 
hands, workable individual differences in the “accuracy” of per- 
mance. There are perfectly distinct differences in the time of 
performance, and there seems little reason, in the present test, 
for extending the scoring beyond this single factor. 
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In this respect, the range of normal performance in the test 
at the beginning of practise would seem to lie between 100 and 
200 seconds, with an average of 133 for the whole blank. The 
subjects averaging this figure were 20 men and 20 women of a 
similar group. It is worthy of note that the women averaged 
distinctly faster than the men, 123 as against 145 seconds, and 
also somewhat less variable, the m. v.’s approximating 15 and 
22. One of the writers has usually found the sex difference in 
variability in the other direction, and, as is brought out just be- 
low, the difference between this and the succeeding experiment 
was somewhat more marked in the women. In both groups the 
second half of the experiment averages slightly faster than the 
first half, though there are great variations in this respect. 

The function with which the test is concerned is one rather sus- 
ceptible to practise, as anyone may readily discover for himself. 
In a succeeding experiment the average times dropped to 116 and 
134 for the women and men respectively, and the first half of 
the experiment averages a little faster in the women. Much of 
the significance of the experiment depends on the preservation of 
the individual relationships originally indicated. They are better 
preserved in the men, the orders corresponding within 13% for 
them as against 24% for the women, though this is largely due 
to two cases, one of whom rises fourteen places, the other drop- 
ping ten. The function is evidently one whose expression in the 
test can be distorted by incidental factors that are as yet very 
imperfectly understood, and the advisability of the greatest possi- 
ble standardization of the experimental conditions and material 
is only further emphasized. 

For subjects who at the outset evince a fairly definite quality 
of performance, practise does not tend to any great alteration of 
relative position. There seems to be less individual difference 
in susceptibility to practise than in the range of performance at 
the beginning of practise. 

Form B. This variant is termed the “number-group checking 
test.”” It will be borne in mind that the conditions of the above 
discussed experiment have been frequently altered to require the 
checking of more than one symbol. The most familiar form of 
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this is the so-called a-t test, in which the subject, using a passage 
of connected prose, checks every word containing both the letters 
aand¢. Other tests of a similar nature have been known by the 
names of the symbols checked. This is not the same as merely 
requiring the subject to check all of two or even more symbols, 
since the symbols have now to occur in a definite combination. 
The original blank was therefore not adapted to this purpose, 
and in order to meet in an analogous way the possible require- 
ments of such a test, a special form was constructed for it. 

This test contains all the combinations of nine digits taken six 
at a time. The number of such combinations = 9.8.7.6.5. 
4/1.2.3.4.5.6.—==84. Each single digit occurs ,C, == 56 times; 
each pair of digits ,C, == 35 times; each three, ,C, = 20 times; 
each four, ,C. = 10 times; and each five, ,C, = 4 times. 

In preparing the test, a separate card was used for each of the 
84 combinations, which were first written in the order of the 
digits, e. g., 134689, 125678, 245789, etc. Then all the 35 cards 
containing both of the digits 1 and 2 were separated out of the 
pack, and this pair of digits was assigned to the various possible 
positions in the group of six with approximately equal frequency. 
At the end of this operation, therefore, there were, as nearly as 
possible, equal numbers of groups arranged in the following ways 
(the dots indicating positions not yet filled) : 


12 I.2 I..2 I...2 I 2 
I2 I.2.. 2 
I2.. I.2. 
I2. I.2 
21 2.1 2.1. 2 I 2 I 
2I -2..1 2 I 
21 2.1. 
2.1 


Owing to the preliminary shuffling of the cards, the relation 
of these positions of the pair 1,2 to the remaining composition 
of the combinations was haphazard. 


Next all of the cards containing the pair 1,3 were separated 
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out of the pack and shuffled, and the assignment of the pair 1,3 
to its several possible positions with equal frequency was under- 
taken in the same manner as before. The operation of pure 
chance was somewhat limited by the previous assignment of the 
digits 1 and 2. The same operation was repeated with each pair 
of digits, 14, 15, 16, 17, 18, 19, 23, 24, 25, 26, 27, 28. 
29, 34, 35, 36, 37, 38, 39, 45, 46, 47, 48, 49, 56, 
57, 58, 59, 67, 68, 69, 78, 79, 89, in this order. As a matter 
of course, the operation of chance became more and more re- 
stricted as this work progressed; and it became more and more 
difficult and finally impossible to insure that the several positions 
of a pair should occur with equal frequency. All that could be 
hoped for was approximate equality of arrangement of the dif- 
ferent pairs; and a review of the result, at the close of this 
operation, showed that approximate equality had been attained. 
Each pair of digits, therefore, will be found not far from once 
or twice in each of its 30 possible positions. Since it is believed 
that the chief application of this blank will be for checking the 
groups containing a given pair of digits, this approximate equality 
of arrangement of all the possible pairs is probably what is most 
needed. Some care was however taken also to avoid undue re- 
petitions of the same arrangements of groups of three digits. 

After the internal arrangement of each combination of six 
digits had thus been determined, the cards were again thoroughly 
shuffled in order to determine a chance order of the combinations. 
But the order was not left entirely to chance, for the immediate 
recurrence of the same pair of digits in the same position was 
avoided. Such immediate recurrences are likely to be noticed and 
remembered and so interfere with the repeated use of the blank 
with the same pair. ; 

The result of all the operations so far was the obtaining of a 
series of the 84 combinations of the nine digits, taken six at a 
time, with approximate equality in position of each pair of digits, 
and approximately chancé order of the combinations. Now it was 
desired to double the length of the blank, and in such a way that 
the second half of the blank should be equivalent to the first half. 
Each of the 84 combinations was therefore to be repeated in a. 
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new permutation, and the equality in the arrangement of pairs 
was to be maintained. This could be accomplished, without need 
of going through all the operations involved in preparing the first 
half of the blank, by simply permuting each of the groups of the 
first half in the same way. Each of them was, in fact, permuted 
according to the following scheme: 2, 4, 6, I, 3, 5. The second 
set of 84 groups thus obtained must have all the characters im- 
pressed on the original set, only with change of the absolute 
digits ; and there can be no duplication between the two sets. By 
similar schemes of permutation, 720 sets in all could be obtained 
without duplication at any point. 

After the above blank had been printed, it was checked up in 
every way to see whether it was according to specifications, and 
found to be correct. Since, in spite of the statistical equality be- 
tween the halves, there might still be inequalities in practise, due 
to the element of chance entering into the arrangement, the halves 
were empirically compared, by taking the time for each half both 
for each single digit and for each pair. Our results are not, in- 
deed, numerous enough to establish the precise equality of the 
halves, but they give no reason to suppose the halves different in 
any respect. | 

Were the nine digits of equal perceptibility, this blank would 
afford a large number of equivalent tests. But the digits are 
quite unequal in perceptibility; and it is therefore necessary to 
establish the relative difficulty of the several tests by trial. We 
have tried the following tests: (1) cancelling the groups con- 
taining each single digit, 9 different tests; (2) cancelling the 
groups containing each pair of digits, 36 different tests; (3) a 
few of the 84 possible tests in cancelling groups containing three 
assigned digits. The results, though not as extensive as could 
be wished, show much regularity and can probably be taken as 
indicating, approximately, the relative difficulty of the several 
tests. 

(1) Cancelling of groups containing a specified single digit. 
In this, as in all the following results, one half of the blank was 
used at a time, and the time is given in seconds. 

Seven subjects, previously untrained in this test, were tested 
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983642 
426357 
654173 


837162 
458671 
275148 
513978 
197584 
918654 
397841 
872351 
923871 
867314 


963458. 


345962 
672389 
312876 
934612 
954178 
719325 
594231 
349716 
714932 
649752 


168379 
372159 
947386 
691324 
971648 
318495 
182765 
563792 
846975 
961872 
327984 
632791 
462758 
981374 
941258 
346521 
853926 
739548 
371629 
294736 
389254 
427395 
759431 
718254 


694517 
7154936 
589761 
814536 
479612 
635728 
615832 
748315 
453867 
248691 
437528 
765429 
486592 
156843 
182653 
427163 
587436 
843216 
529817 
639187 
196235 
138962 
382145 
596743 


253914 
297835 
134852 
326175 
495683 
596873 
851279 
861395 
281463 
574389 
864712 
235849 
198537 
259671 
561487 
281937 
296851 
215367 
436978 
286415 
825749 
268794 
953624 


862934 


745682 
627519 
146237 
368792 
784295 
982563 
498136 
421856 
213956 
332416 
825916 
672834 
871596 
762491 
435781 
672539 
784623 
916483 
123874 
593182 
461289 
524617 
714529 
851763 


158923 
786531 
194526 
549826 
817243 
431289 
356719 
973124 
651274 
723964 
682543 
295481 
164985 
983567 
179428 
985273 
875126 
294378 
957641 
297568 
378652 
358472 
635819 
329418 


729648 
731469 
936425 
572194 
916328 
381647 
412789 


526987 
473519 
534169 
349257 
247153 
579361 
731825 
956142 
013647 
768914 
682917 
145389 
672841 
319546 
237465 
495867 
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in cancelling each of the nine digits, the order of these nine tests 
being different with different subjects, so that any transferred 
practise effect from one digit to another is to a large extent 
equalized in the average of the seven subjects. 


I 2 3 4 5 6 7 8 9 

43.5 63.0 50.9 53.7 61.4 70.9 54.2 57.4 65.1 
COME 37-50 53-81 46-74 42-63 48-78 62-82 390-67 43-69 50-78 


One subject made eight trials with each digit, showing rather 
slight improvement after the second round. For trials 3-8, his 
times average as follows: 


Eo caer caked 3 I 2 3 4 5 6 7 8 9 
31.0 45.5 40.1 38.7 42.9 50.8 36.1 37.6 44.4 


Much the easiest digit to cancel is 1. It is easiest with every 
subject tested, and in every trial. If the times for the other 
digits are expressed as per cents of the time for the digit 1, the 
following are the relative times: 


Relative time, av. of 7 
unpractised subjects 100 145 138 123 I41 163 125 132 ~=~«150 
Relative time, one 
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practised subject... 100 147 130 125 138 164 116 121 143 
Relative time, 3 sub- 


Av. of above, with 
double weight al- 
lowed for first trial 100 148. 134 125 138 163 122 128 + 146 


The relative times in the three sets of results are in fairly 
close agreement, and the combination in the last line of the 
table can certainly be relied on within a few per cent. It is cer- 
tain that 6 is the hardest digit to find, as 1 is the easiest. The 
important practical question is whether any digits are of nearly 
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equal difficulty, so as to be available for equivalent tests. Ap- 
proximate equivalence is assured for the following pairs: 

4 and 7 

3 and 5 

2 and 9 
Further experience with the tests will probably show the need 
for slight corrections in treating these pairs as equivalent. 

When only one pair of equivalent tests is desired, the easiest is 
probably the best, especially as our results show that errors and 
omissions are less frequent with the digits that give shortest times. 
Thus, the seven unpractised subjects whose times are reported 
above gave the following average number of errors (mostly 
omissions) per test: 


In cancelling for the digit 4, 7 or 8................ WA _ 
In cancelling for the digit 3 or 5.................. I ? 
In cancelling for the digit 2 or 


Since the time measure is of most value when errors are ab- 
sent, the digit 1 is indicated as the best to use, except when there 
is need of an equivalent pair of tests; in that case, 4 and 7 are 
the best to use. 

In regard to a correction for errors, our experience has not 
shown the need of one. Our subjects have not seemed to save 
time by omissions, but the time has been about the same either 
with no errors or with one or two or even three omissions. These 
subjects were, to be sure, serious and attentive; and it is likely 
that a more varied experience with the test would show the 
desirability of correcting for errors. We judge that the correc- 
tions should be small, and suggest the addition of 2 per cent. of 
the subject’s time as penalty for each error or omission,’ when 
one half of the blank is used; or 1 per cent. when the whole 
blank is used. 

(2) Cancelling of groups containing a specified pair of 

*An expeditious method of detecting errors is afforded by a key on trans- 


parent paper, to be laid over the blank. Whichever digit is used, the number 
of groups to be checked is 56 in each half of the blank. 
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digits. One subject has made six trials of each of the 36 tests of 
this sort; and another subject has made one trial of each. The 
results appear to have sufficient regularity to indicate the relative 
difficulty of the several pairs, and to show something regarding 
the mental process involved in this form of test. 

The time occupied in checking a pair of digits is always longer 
than the time for checking either digit alone, but less than the 
sum of the times for checking the digits separately. For example, 
a subject takes 42 secs. to cancel the digit 4, and 48 secs. to 
cancel g; to cancel groups containing both 4 and 9 takes him 64 
secs., which is 71 per cent. of the sum of 42 and 48. The time 
for a pair is closely correlated with the sum of the times for the 
digits of the pair, and is usually equal to about 70 per cent. of this 
sum. The results are condensed into the following table. 


Ratio, IN Per CENTS, OF THE TIME FOR CHECKING A PAirR oF DIGITs To THE 
SuM OF THE TIMES FOR CHECKING THE Dicits SEPARATELY 


Subject Average P. &. A. D. Range 
67.3 0.4 3.1 60-75 
72.2 0.6 4.5 62-82 
R. S. W. after practise........ 72.5 0.4 3.0 66-79 


Since this “ratio”’ is fairly uniform, it can be used, in connec- 
tion with our previous table of times for checking single digits, 
to indicate the approximate times for checking pairs. Equivalent 
tests can be selected in this way; among the tests which appear 
from all our present results to be nearly equivalent, we recommend 
the following two pairs: 23 and 89. These have the advantage 
of not conflicting with the digits 4 and 7 recommended for use 
when single digits are to be cancelled. The time for either pair is 
about twice that for the single digit 1, or about one-and-a-half 
times that for the single digit 4 or 7. 

It is possible, from comparison of the results of the two sub- 
jects in the above table, that there are genuine individual differ- 
ences in the “ratio,” i. e., in the speed of cancelling pairs as com- 
pared with the speed of cancelling single digits. Such differences 
may however, be merely the result of the relative degree of prac- 
tise in the two sorts of test. If the subject has gone further in 
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his practise with pairs than in his practise with single digits, the 
ratio will evidently be small. If he has had some practise with 
single digits, but none with pairs, his first experience with a pair 
is likely to give a high ratio. Thus, subject R.S.W., after making 
two trials with each of the single digits, proceeded to try in 
succession each of the 36 pairs. 

The average “ratio,” for the successive quarters of this series, 
was as follows: 


Av. A. D & 
67.9 3.2 0.9 


He then resumed practise with the single digits, and after- 
wards returned to the pairs, obtaining then the average ratio of 
72.5, as shown above. If practise is continued pari passu with 
single digits and with pairs, the ratio would probably remain in 
the neighborhood of 70 per cent. But at the very start, the time 
for a pair is likely to be about 77 per cent. of the sum of the 
times for the single digits; this is indicated also by less complete 
results from several other subjects. 

The dependence of the “ratio” on practise has a bearing on 
the theory of this test. The fact that the ratio is high at the first 
experience in cancelling for a pair of digits shows that the de- 
tection of a pair of digits in a group is a specialized performance, 
not reducible to the acts of detecting the single digits. The de- 
tection of any specified pair of digits is no doubt a specialized 
performance, susceptible of very special training; this has indeed 
been shown in similar cases by Thorndike and Woodworth. But 
in the present series of tests, the pair of digits cancelled was 
changed with each new trial, so that the training visible in the 
lowering of the ratio from 77 per cent. to 72 or 70 per cent. is an 
example of transferred practise, and indicates that there is some 
element of skill common to the checking of all the pairs of digits. 

Though the ratio varies within rather narrow limits and shows 
a comparatively small A.D. (as seen in the table on p. 36), 
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yet there is sufficient variation to make it possible that the ratio 
varies according to the nature, difficulty, etc., of the pair em- 
ployed in the particular test. We have been unable to find any 
characteristic difference, however, according to the difficulty 
of the digits entering into the pair, except that, in case of subject 
R.S.W., the ratio is low for pairs containing the digit 1. The 
subject, after practise, gave the following ratio for pairs contain- 
ing the several digits. 


Pairs containing the digit Average ratio P. E. A. D. 
Ter 68.2 0.5 1.7 
73.7 0.7 2.3 


In general, the ratio seems not to depend on the digit; and 
the same negative result appears in case of the other subject, 
J. W. T., who moreover does not show anything characteristic 
of the digit 1. But the above tesults from subject R.S.W. show 
an unmistakeably lower ratio for pairs containing the digit I. 
The distribution is fairly bimodal, the pairs containing 1 forming 
a group by themselves. | 

Some explanation of the low ratio for pairs containing the 
digit 1 is afforded by R.S.W.’s introspective account. It early 
occurred to him that a good device for cancelling groups con- 


taining a pair would be to look first for the easier digit of the 


pair, and thus to look for the harder digit only in the groups 
where the easier digit appeared. In practise, however, this device 
did not seem to him to work very well, except when the easier 
digit was 1; he tried to use the device also when the easier digit 
was 4, 7 or 8, but without subjective indications of success. 
When one of the digits was I, groups containing it could be 
recognized in indirect vision, and thus many groups could be 
passed over altogether in direct vision. Subjectively, this method 
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of working required more effort but appeared successful. The 
objective records, as crystallized in the “ratio,” show that the 
device was a Success in the case of the digit 1. 

Further consideration of this point may throw some light on 
the mental process involved in this test. If finding a pair were 
the same thing’as finding the members of the pair, with no over- 
lapping, the time for the pair would be the sum of the times for 
the digits composing the pair—instead of being, on the average, 
only 70 per cent of that sum. There must therefore be considerable ° 
overlapping or condensation. On the motor side, there is a pos- 
sible condensation of the checking movement, but this is so quick 
and automatic anyway that abbreviating it has probably little to 
do with the shortening of the time. More strain is probably put 
on the eye movements when the speed of the work approaches its 
maximum (about 3 groups covered per second) ; but since this 
maximum is not approached, in our results so far, except by one 
subject in case of the single digit 1, the probability is that the de- 
mands made on the eye are well within its motor capacity. The 
difficulty of these tests is mainly perceptional, and the over- 
lapping which is effective in finding pairs of digits must occur in 
the perceptive process. 

If the device described above as adopted by one subject in find- 
ing pairs of digits—a device which has frequently been adopted 
by other subjects in similar tests—if this device represented the 
essentials of finding the pair quickly, then the following calcula- 
tion should hold good. The subject looks first for only one digit, 
and where he finds it looks for the other one. The task of looking 
for the second digit would be necessary only in % of the total 
number of groups in the blank (since the first digit, or any digit, 
is present in 56 out of the 84 groups). If therefore this plan were 
carried out systematically and without hitch, the time for checking 
a pair should be equal to the time for checking the first digit plus 
¥%3 of the time necessary to check the second digit in the entire 
blank. For example, in checking the groups containing both 1 
and 2, the time would be that needed to find the 1’s (and this is 
31 secs.) plus 2% of the time necessary to go through the blank 
for 2, namely % of 45.5 seconds, or 30.3 seconds; which added to 
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31.0 secs. gives 61.3 secs. as the calculated time for checking the 
pair 12. But the observed time is considerably less than this, 
namely 53 secs. ; and this same discrepancy between the calculated 
and observed values obtains in every instance. The time for 
checking a pair is never as long as it would be if the above 
device were followed systematically. ‘ 

This device does not therefore constitute the essential mechan- 
ism of checking a pair of digits. The device seemed to work 
‘well with one subject, in case one of the digits was 1; but its 
conscious use only reduced the “ratio” from 72.5 to 68.2, or 4 
units, whereas something else reduced the average pair from 
the maximum of 100 to 72.5, or 27 points. There must ac- 
cordingly be some form of overlapping of which the subject is 
not clearly conscious, but which is much more efficacious than 
the best devices which he consciously adopts. Introspection gives 
some hints at.such an overlapping. Sometimes, indeed, a group 
is successively examined for the two digits composing the pair; 
but this is rather the exception. Often the presence of both 
digits is simultaneously recognized; and still oftener the absence 
of the pair is recognized without a clear consciousness of which 
of the two digits is lacking. 

(3) Cancelling of groups ‘containing three specified digits. 
Our data here are limited to 25 tests with one subject. The time 
occupied in this test is, again, closely correlated with the sum of 
the times for the three component digits, and is equal to about 
70 per cent. of the sum of these times. Apparently the ratio is 
slightly lower for three digits than for a pair, for the subject, 
R.S.W., gives an average ratio of 68.3 per cent., with A.D. of 
5.4, and P.E. of the average of 0.9. This average is thus prob- 
ably lower than the average of 72.5 obtained by this subject with 
pairs. 

When 1 is one of the three digits, the average ratio is 62.0, 
A.D. being 3.4 and P.E. 1.1. The subject adopted the same 
device as in pairs containing the digit 1, and the results here are 
confirmatory of what has previously been said. 

For two approximately equivalent tests, we recommend can- 
celling for 146 and for 257. The time for each of these is about 
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2.5-3 times that for the single digit 1, or about 2-2.5 times that 
for the single digit 4 or 7. 

Use of the number-checking blanks with laboratory classes. 
As suggested above (p. 27), the number-checking blanks could 
readily be adapted for experiments in continued mental work, in- 
terference, etc. One of the writers has used Form B, the ““Num- 
ber-group Blank,” with success in an experiment in practise and 
“transfer.” Half of the blank being used as the unit, the sub- 
ject first checked the groups containing the digit 6, then took a 
practise series of ten units with the digit 7; then one unit again 
with 6; then another practise series with digit 4; then one unit 
with 7 and finally one more with 6. The two methods of studying 
“transfer,” namely the “cross-section” method, and what may be 
called the “‘successive practise curve’? method,’ are combined in 
this experiment. The tests with digit 6 give cross-sections before 
and after practise with other digits; and since the digits 4 and 
7 are equally hard to find, the practise curve with the one, follow- 
ing that with the other, should show the effects of the preceding 
practise. Transfer is pretty sure to be in evidence in each stu- 
dent’s results; these need, to be sure, some correction from con- 
trol experiments in which the cross-sections are taken without the 
intervening practice.” 


‘Introduced by Bair, “The Practice Curve”, Psychol. Rev., Monogr. Suppl. 
No. 19, 1902. 
*See W. F. Dearborn, Psychol. Bulletin, 1909, 6, 44. 
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IV. ADDITION TESTS 


For rigidity of associative control, no experiments surpass 
those involving the simple arithmetical processes. In these a cer- 
tain arithmetical task is visually presented to the subject, and 
efficiency is measured in terms of time and error. To provide 
an objective criterion of the performance the subject is required 
to speak or write the result. According to what has gone be- 
fore, oral response is employed exclusively in the present experi- 
ments. One advantage of the written response is thus dispensed 
with, namely the permanency of the record, through which to 
check its accuracy. This difficulty is best obviated through pro- 
viding the operator with a key upon which the correct reactions 
are noted. The operator follows the responses of the subject on 
the key and so keeps account of the data to be recorded. 

Such experiments with arithmetical processes have an almost 
infinite range of difficulty, varying in practise from the simple 
addition of a pair of digits to the mental multiplication of three 
and even four place numbers. The chief advantage of the former 
is their freedom from errors; of the latter, the greater proportion 
of time spent in the: essential work of the test. At first glance, 
one might consider that this same consideration, which leads to 
the substitution of oral for written response, should lead to 
the rejection of the easier and adoption of the more difficult 
experimental material. But it were very easy to press this ad- 
vantage too far, especially in tests that are intended for any- 
thing like general employment. The more complex intellectual 
associations would result in the average individual in an im- 
possible number of errors, if indeed they did not prove too much 
for his patience as well as his powers. A test not intended for 
limited application should not be one limiting the subjects who 
can respond to it; the tests to be described here, therefore, deal 
with the simpler arithmetical processes, regularly of addition, 
though the material prepared is adaptable in various ways. 

1. The particular form of addition test with which the most 
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work has been done, and whose properties with reference to the 
work curve are best understood, is that of the Kraepelinian 
Rechenhefte. This is a pamphlet of twenty-four pages, upon each 
of which are printed nine vertical columns of 32 single digits in 
apparently random succession. It is possible to experiment with 
this material by continuously adding the successive digits, and 
announcing the sum total at stated points. The disadvantages of 
this procedure are very numerous and do not call for considera- 
tion here. A decidedly preferable method is the simple addition 
of the successive pairs of digits. That is, the first four figures in 
the Rechenheft being 8, 3, 5, 7 the sums announced by the sub- 
ject are 11, 8 and 12. The subject continues to announce the 
sum of every figure plus the one next below it. Precise control 
of the whole process, both as to accuracy and time, is thus secured. 
As before mentioned, the operator checks the correctness of the 
sums, notes errors, and the amount performed within specified 
times. The usual periods of work with the Kraepelinian test 
have been of five and ten minutes each, and it has also been 
customary to record the amount of work done during the single 
minutes. The subject should not, as has been done, be called upon 
to make the records; all such tasks devolve properly upon the 
experimenter. It does not appear that a significant portion of 
the time is consumed in the motor process of response. At the 
beginning of practise, the number of additions made in five min- 
utes is usually under two hundred. It need scarcely be said that 
the sums themselves could be read in a much shorter time; maxi- 
mum speed of reading aloud in normal individuals averages not 
far from 100 words in 30 seconds. 

Used in the above way the Kraepelin Rechenheft contains 31 
additions pes column, 279 per page. While the unpractised sub- 
ject is not likely to do more than this in five minutes, a little 
practise will soon take him over the page, and it may be con- 
sidered always advisable to open the Rechenheft to two full pages, 
(it should be held open with a clip, amply providing for any com- 
mon performance without turning a leaf. One of our subjects, 
after prolonged practise, occasionally reached a figure above 558 
additions in the five minute period. There is uniformly a con- 
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siderable practise effect in the test, perhaps due partly to the 
unfamiliar sort of reaction required to the material, but, as with 
the number-checking test, the individual differences in suscepti- 
bility to practise are small in comparison to the differences in the 
amount of work performed. 

The Kraepelin Rechenheft is practically the only form of this 
experiment that permits long continued tests of the same subject 
with sufficient uniformity of experimental material; but as the 
present tests are not designed as practise experiments, it was 
thought advisable to construct a more convenient blank for a 
single or small number of determinations. It was also endeavored 
to improve on certain minor features of the Rechenhefte, as the 
odd number of columns and additions in each column. 

The blank, reproduced on the page opposite, contains 24 
columns of 26 digits (25 additions) ,—in all, 600 additions. This 
is probably beyond the 5-minute capacity of the normal subject at 
the limit of practise. The columns are arranged in six groups 
of four, each thus containing 100 additions with 104 digits. The 
scheme of the distribution of the digits was simpler than in the 
number-checking test. 99 slips were prepared, eleven for 
each of the nine digits. The fiye remaining slips bore the digit 5. 
Random drawings were made! from the group of slips, and the 
digits were written in the columns in the order in which they 
were drawn. Each set of four columns (100 additions) is there- 
fore a unit in itself, and is made up of a proportionate number 
of each of the nine digits in random order. The six series of 
100 additions may then be considered as homogeneous and of 
approximately equal difficulty. 

Subjects are not apt to comprehend this test so readily as they 
do, for example, the number-checking test. They “may tend to 
add the numbers continuously, or more especially to add discrete 
numbers, thus with the column beginning as before, 8, 3, 5, 7, to 
announce the sums II and 12, omitting to add the 3 and 5. The 
operator must be well assured from the instructional material that 
the test is properly comprehended. Some subjects prefer to fol- 
low the columns with the finger; this should be permitted, on the 


same principle that permits reversed directions in the number- 
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checking test. The addition of the final digit of one column to the 
beginning one of the next is not required, nor do subjects ordin- 
arily tend to do it, even though not specially instructed. 

The test may of course be made of any length within the limits 
of the blank, but, from experience with the Rechenhefte one hun- 
dred additions appear sufficient for a unitary determination. 
The length of the present blank is believed to provide amply for 
sufficient variation of the experimental material for all ordinary 
purposes outside of special research upon the individual function. 

The test requires the constant attention of the operator to the 
key to check the proper performance of the work by the sub- 
ject. False reactions are more frequent than in the number- 
checking test, and they should be kept track of, though the writer 
has not seen an instance where they obscured individual differ- 
ences in efficiency. The subject may not notice the error; if he 
does, the purpose should be to get him over it with as little dis- 
turbance and distraction as possible, and he should therefore be 
allowed to correct it or not, whatever is the path of least resis- 
tance for him. We believe this to be the sounder experimental 
practise, whatever might be said of it from an ethical standpoint. 

2. In order to furnish a regulated experimental material 
which should have a greater flexibility of application than is 
usual in this class of tests, a second form of procedure is sub- 
mitted, known as -the constant increment test. This is a little- 
recognized method, but one which in direct comparison has shown 
superiority over other forms. It consists in presenting to the 
subject a series of numbers, requiring the identical arithmetical 
operation to be performed upon each. In the observations made 
with this test, the usual procedure has been the addition of 4. 
In this particular instance, there is perhaps no reason why the 
Kraepelin blank should not serve this purpose as well as that for 
which it is ordinarily used. In order however, to make the ma- 
terial adaptable also to subtraction, especially of larger figures, it 
is thought wise to preserve the special blank originally adapte 
to this test. | 

This blank contains 100 two place numbers. The unit places in 
these numbers are ten each of the figures 1,2,3,4,5,6,7,8,9,0. In 
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the tens places are ten 2’s, ten 7’s and twenty each of the interven- 
ing 3,4,5 and 6. These features are symmetrically distributed with 
reference to the halves. The subject speaks the proper responses 
according to the assigned experimental task; thus in the addition 
of 4 the responses begin 68, 53, 66, etc. It is probably advisable 
to confine the unit of observation to half of the blank, and even 
better would probably be two tests of one column each. Errors 
seem to be but slightly more frequent than in the simple addition 
of a pair of numbers; in their treatment, the same considerations 
obtain as in the previous form of test. Since the operation to be 
performed with the given numerals may vary indefinitely, no 
key is provided, but the experimenter may readily provide one 
himself for his own particular requirements, and should always 
do so; its employment being the same as in the Kraepelin form 
of addition test. 

Results. Individual differences, due in part, no doubt, to dif- 
ferences of training, are very great in even the simplest arithme- 
tical tests. Thus, while one of the authors has usually obtained, 
with the Kraepelin form of test, times of from two to three 
minutes for 100 additions, the other of us, working on 7 college 
and university students, has the following results: 


Average time for 100 additions...... 107.2 seconds 


With the constant increment test, the following results have 
been obtained from 10 subjects of the same class as above: Only 


one column was used in each test, and the times given are times 
for one column. 


Add 4 Subtract 4 Add 17 
Av. errors per column of 25.. 0.3 0.2 2.4 


Experience with other subjects leads one to suspect that the 
time per column, for adding 4, will often run up above 60 seconds. 
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V. NAMING TESTS 


1. The Color Naming Test. No doubt the best-trained and 
quickest of all associative responses appear in the reading of 
words, letters and numbers. (See above p. 9). Next to this, 
probably, stands the naming of familiar objects. The color- 
naming test belongs here, and may be expected to give shorter 
times, per reaction, than any of the other tests included in the 
present selection. 

The test blank shows 100 patches of color (besides 5 sample 
patches), each I cm. square and separated by spaces of I cm. 
from its neighbors. The colors are to be named in order, as in 
reading, and one side is indicated as the top. The arrangement 
is such, however, that the blank can be equally well used in any 
one of the four positions. 

Care is needed in the selection of colors for such a test; for 
the color names required should be of universal familiarity, and 
there are few such names. White, black, gray, red, yellow, green, 
blue, brown, and possibly pink and purple are in sufficiently 
general use for our purpose. It seemed safer to exclude pink and 
purple; if white is then used for the background, seven colors 
remain; but it seemed better, on the whole, to employ only five 
stimulus colors, repeating each twenty times. After some experi- 
menting, it was decided to use red, yellow, green, blue and black. 

The use of the Hering red and green seemed undesirable, 
since they do not suggest the color names so promptly as do a 
red and green each somewhat nearer to yellow. To attempt to 
equate the colors in brightness would not be best, since this would 
mean toning down the yellow to a point at which it would be 
likely not to be called by that name. The blue must be pale in 
order to be distinguished ‘from black as quickly as the other colors 
are distinguished from each other. 

In the arrangement of the color stimuli on the blank, all 
sequences of the same color were avoided, as such direct sequences 
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enable the subject to combine two reactions, practically, into one. 
Too frequent recurrence of the same sequence of different colors 
was also avoided, the object being to compel a separate reaction 
to each single stimulus. It was desired, indeed, to have the dif- 
ferent possible sequences of two and of three colors occur with 
equal frequency ; but something less than perfection in this regard 
had to be accepted, for it was desired also to make the arrange- 
ment equally good for each of four positions of the blank; and 
all the conditions of perfection could not be met in both the 
horizontal and vertical lines at the same time. An incredible 
amount of time was consumed in arranging the colors to meet 
these simple requirements. The blank seems now to be free 
from serious blemishes of arrangement, and requires approxi- 
mately the same time for reading in each of the four positions. 

Preparatory to the test, the experimenter lays the blank before 
the subject, with only the sample line of 5 colors showing. The 
subject is directed to give the names of the sample colors; when 
he understands the task and knows what names to use, the whole 
blank is exposed at the word “Go!” The time for the first 
half, as well as for the whole blank, may well be taken. Re- 
peated trials can be made with the blank in four positions. The 
line of 5 sample colors is to be omitted in the actual test. As 
usual, one half of the blank is long enough for a satisfactory 
test. A key of the series of correct responses will facilitate the 
experimenter’s task. 

2. The Form Naming Test. The blank next to be described 
under the head of the Substitution Test is similar in all respects 
to the color sheet, except that five geometrical figures take the 
place of the five colors. This blank can be used in the same 
manner as the color blank, the numbers written in the key line 
being disregarded. 

Results. Here, as in most of the other tests, the results now 
available are insufficient to do more than give a general impression 
of the time required. In‘this case, too, the results are all from 
one class of subjects, namely college and graduate students. The 


whole blank of 100 stimuli was reacted to, and the time taken for 
the half as well as for the whole. 
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Color naming test. 
Ist half 2nd half Whole 


Total range, both sexes together....... 22-41 24-48 48-89 


Form naming test. 
Ist half 2nd half Whole 


- 46.7 47.2 93.8 

6.0 10.0 16.0 
2.5 4.2 6.7 

Total range, both sexes together.... 31-60 29-58 60-117 


From these data, it may be inferred (1) with reasonable as- 
surance, that the color naming test is easier than the form-nam- 
ing test. Comparison of the figures for the present color-nam- 
ing test with those reported by Wissler’ from the same class of 
subjects with the use of the Columbia color-naming test, which 
employs ten colors, makes it probable that the present test with 
five colors is noticeably easier—as was, indeed, intended. 

(2) It may be inferred from the above table, with much 
probability, that a sex difference exists in the case of the color 
naming test, women being on the average quicker than men. 
This is the more probable because Wissler? obtained the same 
sex difference from much more extensive data. 

(3) It seems also probable that the same sex difference exists 
in the case of the form-naming test. If so, the sex difference 
here in question is not specially related to the color sense, but 
rather to linguistic facility. The authors have in mind the ac- 
cumulation of sufficient-data to determine whether the appear- 
ance here shown corresponds to a real sex difference. 

(4) In the color-naming test it seems probable, and in the 


* Psychol. Rev., Monograph Suppl. No. 16, 1901. 
Op. cit. 
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form-naming test, possible, that there is some slackening of re- 
actions, such as was shown above (p. 15) to occur in other forms 
of test. On reference to the individual records, we find that 10 
of the 14 subjects in the color-naming test took longer for the 
second half than for the first half of the test; and in the form 
naming test, 6 out of 10 did the same. Only one of the ex- 
ceptions is more than a bare exception. The behavior of the 
subjects during the test shows periods of hesitation and ob- 
struction, and even of false reaction— rather a strange phenome- 
non, in view of the great familiarity of the names and their 
correct use immediately before. The subject is aware of this 
inhibition, and it is a strange experience for him. The “mechan- 
ism” of inhibition can not here have the elaborate Freudian char- 
acter; and in fact the experiment seems a good one to show the 
reality of other forms of inhibition in the recall of names. The 
real mechanism here may very well be the mutual interference 
of the five names, all of which, from immediately preceding use. 
are “on the tip of the tongue’, all equally ready and therefore 
likely to get in one another’s way. These periods of inhibition 
do not appear at the very beginning of the test, but most often, 
to judge from incidental observation, along in the middle. Some 
subjects, after succumbing a few times to interference, appear 
to collect themselves and do the last part of the test better than 
the middle. We haye the records of the successive rows of ten 
stimuli each, in the case of five subjects in each test. The aver- 
age time in seconds for each row is as follows. 


oss cows I 2 3 4 5 6 7 8 9 10 
Color naming ... 62 56 56 66 68 73 77 69 63 6.2 
Form naming ... 80 91 90 103 108 82 105 9.9 103 9.0 
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VI. FORMATION OF NEW ASSOCIATIONS 


The Substitution Test. This blank is modelled after one by 
Professor J. E. Lough,’ but is simplified, in that only five (instead 
of twenty) different stimuli are used; at the same time, by em- 
ploying geometrical forms in place of the letters of Professor 
Lough, it is partly freed from the danger that some subjects may 
hit upon easy mnemonics. 

Since the names of the forms may enter into the subject’s pro- 
cedure, the forms should have equally familiar names. They 
should also be of such shapes that the blank, like the color-nam- 
ing blank, may be capable of use in different positions. Only 
about five geometrical forms meet these conditions: the circle, 
square, triangle, star and cross. The blank is made up of these 
five forms, each repeated twenty times. The arrangement of 
the stimuli follows the same rule here as in the color-naming 
test. 


At the top of the blank appears a line containing each of the 


‘five forms once, with a number on each. This line being cover- 


ed, the rest of the blank is exposed to the subject, and it is ex- 
plained to him that he is to write on each of these forms a num- 
ber,— the same number as he will find on that form in the key 
at the top. In this test, the general rules of instructing the sub- 
ject by aid of examples can not be exactly followed; for the as- 
sociation to be employed in the test should not be formed before 
the outset of the test itself, since this is a test of the formation of 
associations. When the experimenter is sure that the subject 
understands what is to be done, he uncovers the key, at the word 
“Go!” Besides taking the time for the whole and half, the ex- 


perimenter may be able td get the times for each successive line, 


and so obtain a curve of the formation of the associations. 

A misunderstanding which has occasionally appeared in the 
use of this test should be guarded against in the instructions to 
the subject. Some subjects have started to go through the 
blank numbering only one of the forms at a time, intending to 


* Described by Kirkpatrick, Studies in Development and Learning, Arch. of 
Psychol., 1909, No. 12, p. 36. 
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go through again for each of the other forms. It should be 
made clear that the forms are to be numbered in order as in 
reading or writing. . 

Results. Eleven educated adults (6 men, 5 women) gave the 
following average time in seconds: 


Ist half 2nd half Whole 


53-83 111-177 


The gain from the first half to the second is perhaps not so 
great as would have been expected. In fact, few if any of the 
subjects fully mastered the key in the course of the 100 reactions. 

Time was taken, in this test, for each successive row of ten 
forms, with the following average results (11 subjects) : 


a ee I 2 3 4 5 6 7 8 9 10 


The longer time for the first line than for the second is found 
in 9 of the 11 individuals; and 3 subjects do the first line as 
quickly as any other. It is, in short, possible to do the first line, 
by simple use of the key, in 9-11 seconds, and this is as rapid as 
any of the subjects became during the course of the test. Some 
subjects do the first line by mere copying from the key; others 
start to memorize and take longer on this line; this is probably 
the cause of the extra large variability for the first line. 

The test is not long enough to permit the complete establish- 
ment of the associations; several blanks may be used in succes- 
sion, and oral may be substituted for written responses in order 
to simplify the motor part of the performance. At the best, how- 
ever, progress is rather slow; and, indeed, one would not expect 
these freshly formed associations to surpass readily the familiar 
associations involved in the form-naming test (p. 51), the times in 
which are, after all, not very much shorter than those in the 
last rows of the substitution test. 
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By use of a second key, the blank can be used for the study 
of interference; one of the authors has so used it in laboratory 
classes, following the general arrangement of Bergstrém’s card- 
sorting experiment.’ The blank can also be used for a simple 
cancellation test, similar to the number-checking tests. 


* Amer. Journ. of Psychol., 1893, 5, 356. 
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VII. LOGICAL RELATIONS 


The form of test in which the stimulus is a word and the 
response another word standing in some assigned logical relation 
to the stimulus has been long and widely used, and has an intellec- 
tual atmosphere that makes it seem likely to prove a test of indi- 
vidual differences of the intellectual sort. At the same time, it is 
distinctly a test of the command of language, and when the 
measurement concerns the speed of the response, familiarity with 
the necessary words is a prime necessity. Unless great care is 
used in the selection of stimulus words, long reaction times will 
occur from the need of searching for the proper response words, 
and the test thus becomes predominantly linguistic in nature. 
Linguistic it must always remain to a considerable extent, no 
matter how much care is taken in the selection of the stimuli; but 
the effort should be to minimize the linguistic factor by selecting 
only stimulus words that are universally familiar. 

Besides the familiarity of the associations employed, the test 
calls for skill in the handling of these associations; and it is this 
skill, most of all, which the test designs to measure. In other 
words, it is the efficiency of the “determining tendency”, or ad- 
justment to react according to instructions, which should be re- 
vealed by the speed of performance. The more completely this 
adjustment dominates the performance, facilitating the right re- 
sponses and inhibiting other, interfering associations and per- 
severations, the less hesitation and confusion will occur and the 
more prompt will be the reaction. 

In order to afford sufficient opportunity for the determining 
tendency or adjustment to show its efficiency, it is customary and 
evidently desirable to provide a number of stimuli in succession, 
requiring the same sort of response to each. There should 
therefore be a list of stimulus words for each of the logical re- 
lations along which the reactions are to be required Thus the 
task of providing material for these tests consists in discovering 
a sufficient number of stimulus words of the requisite familiarity. 
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Only by actual trial can the suitability of the stimuli be ascer- 
tained. A word of apparently eminent fitness may prove to be 
unfamiliar to many subjects. For example, the word “false” 
seemed likely to be a good stimulus when the required response 
was a word of opposite meaning; but in practise much hesitation 
and uncertainty of reaction appeared in the responses to this word 
(instead of “true”, some women subjects said “natural”). A 
word which seems perfectly familiar to the investigator may be 
unfamiliar to many subjects, and a word which seems perfectly 
unambiguous may convey an unexpected meaning to some sub- 
jects. To avoid all such difficulties with all subjects is too much 
to hope; but the test material should be freed from words that 
cause difficulty to a large share of the subjects. This is necessary 
at any rate if the time is to be taken, not for each separate re- 
sponse, but only for the whole series of responses to the list of 
stimuli; for otherwise the total time may be determined mostly by 
the difficulty of one or two of the reactions. Even if the times of 
the separate reactions are taken, the average time will suffer in 
the same way as the total time in the preceding case. The median 
time is mostly free from this source of error. But even so, lists 
of nearly uniform difficulty would form the best and fairest test 
material. 

Our procedure in selecting stimuli for this class of tests was 
to start by getting together as large a number of stimuli as possi- 
ble; to eliminate at once all that seemed ambiguous or unduly 
difficult and to try the remainder with a few subjects, timing the 
separate reactions, and eliminating the stimuli that gave the slow- 
est reactions or that proved to be ambiguous or complex-arousing. 
The abbreviated list was tried in the same way with other sub- 
jects, and more words eliminated, till finally it appeared that the 
easiest possible list of stimuli had been secured. Unless it proved 
possible to secure a list of twenty easy stimuli, that particular 
test was abandoned. Thus, it seemed impossible to prepare a list 
of twenty words sufficiently easy for a synonyms test, except in- 
deed for well-educated individuals. On the other hand, it ap- 


peared possible to select two lists of twenty for the opposites 
test. 
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The number of subjects employed in reaching the selection of 
stimulus words was greater in case of the opposites test than in 
the others, and this test may properly be regarded as more highly 
“standardized” than any other belonging under the head of logical 
relations. In the case of opposites, a long list of words was tried 
with 6 subjects, and the forty words selected from this trial 
were tried with 40 other subjects; some need of revision was 
then apparent, and a few more words were substituted from 
tests of a few subjects; then the revised collection of 40 was 
tried with thirteen fresh subjects, and a few minor corrections 
still introduced, which left the lists in their present condition. 
In the other tests, two to three times the desired number of stim- 
ulus words were tried with nine subjects, and the resulting selec- 
tion was tried with thirteen fresh subjects; some minor changes 
were then introduced and the lists left in their present condition. 


The “mixed relations” test was selected gradually on the basis 


of results from fourteen subjects. 

After the selection of the stimulus words came the question 
of their arrangement within the list. This matter of order of 
stimuli is not of great importance if the time is to be taken for 
the separate responses; but whenever the time is taken only for 
the series, the order of stimuli is a matter of some consequence. 
We recommend, it may be remembered, that the time be taken 
for the first half of the list as well as for the whole list, and even 
that the halves be given as separate tests ; it is therefore important 
to have halves of equal difficulty. Moreover, many investigators 
find it convenient to allow a fixed time for each test, and to 
measure the number of responses that can be given in this time 
(“time limit method’’) ; with this procedure it is important that 
the list shall be of uniform difficulty throughout, so that the 
number completed shall be a fair measure of the work done. 
Whatever be the procedure in giving the test, the most desirable 
arrangement of the stimulus words would be such as to distribute 
the difficulties evenly throughout the list. If it were really pos- 
sible to discover twenty stimulus words of equal difficulty, the 
question of their arrangement would not arise; but this is not 
possible, for though the twenty stimuli be all decidedly easy, yet 
the reaction time to one will be two or three times as long as to 
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another of the twenty, on the average of as many as ten sub- 
jects. Since it is impossible to prepare a list of twenty stimuli of 
equal difficulty, we combined the words in pairs so that the pairs 
should be of equal difficulty, as judged by the sum of the re- 
action times to the two members of each pair. One pair thus 
may consist of the hardest and the easiest word in the list, and 
another pair of two words of medium difficulty; but the sum 
of the reaction times for the first pair is equal to that for the 
second pair, as judged from the records already in hand. (The 
pairs can not be hoped to be equal for all subjects.) These 
equal pairs can then be arranged in such a way as to avoid “con- 
stellations” or undesirable collocations of any sort; and the dif- 
ficulties of the list will be pretty evenly distributed. 

Two other points were considered in arranging the order with- 
in the lists. When the test is given with a time limit, it is es- 
pecially desirable to have the responses of uniform difficulty in 
that part of the list where most of the subjects will be stopped, 
so that there, at least, the single words shall constitute equal units. 
We have therefore placed those of our “pairs” which are com- 
posed of words of medium difficulty in the midst of the list, 
from about the 8th to about the 16th word. If then the time 
limit is so chosen that the great majority of subjects shall be 
stopped in this part of the list, the separate words may, without 
much error on the average, be counted as equal units. 

The other point concerns the writing of ‘responses. In reality, 
as explained in the introduction, an easy association test is very 
ill adapted for written responses, because the time of writing 
is much greater than that of easy association, and individual 
differences in speed of writing altogether mask the differences 
in speed of association. However, in case of the opposites test, 
we have determined the writing times for the correct response 
and so distributed the stimuli that the writing times for the two 
lists of twenty, for the halves of each list, and, as nearly as possi- 
ble, for the pairs throughout each half, shall be equal. This has 
not been attempted for the other tests given below, because the 
response words are not wholly determined in advance. 

1. The Opposites test. This test has one advantage over all 
the others in the series of logical-relation tests, namely that the 
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answers can be definitely scored as right or wrong. Opposites 
apparently are the most available material for a test of completely 
controlled association—with the exception, indeed, of the naming 
tests and of the arithmetical tests already brought forward. For 
this reason, we have taken unusual pains with the selection and 
arrangement of material for this test. As it appears possible to 
select forty words free from difficulty, we offer two lists of 
twenty, with the object of making it possible to give two equally 
difficult tests of the same function. Since, however, it may be 
desired in some instances to have the very simplest material, we 
also present a list of twenty “Easiest Opposites”, all of which 
are included in the two lists of twenty. 

The lists are printed on three separate slips,’ in 12-point type, 
well-leaded. Lists I. and II. are of equal difficulty, and the 
halves of each list-are equivalent, as far as can be judged from 
the results of their use so far. 

The instructions, enforced by samples (see p—), require the 
subject to respond to each stimulus word by the word having the 
opposite meaning; as, “long-short”. 


OPPOSITES TEST 


I IT Easiest 
long north high 
soft sour “> summer 
white > out - out 
far weak white 

Sup good slow 
smooth after yes 
early ™> above above 
dead sick north 
hot slow top 
asleep large wet 
lost good 
wet dark rich 
high ‘front up 
dirty ~ love front 
east tall long 

>day open hot 

summer east 
wrong new day 
empty ‘S come big 

Stop male love 


*The type, etc., of these tests is not reproduced here. 
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2. The verb-object test. More good stimulus words are avail- 
able here than in any similar test except that for opposites. Here 
again we have selected two equivalent lists, and also a list con- 
taining the very easiest stimuli, as judged from results with about 
20 subjects. The verbs are to be treated as transitive, and ob- 
jects supplied; for example, “sing song”, “build house.” The 
audible repetition by the subject of the stimulus word is not re- 
quired, and may interfere somewhat with the experimenter’s re- 
cord; but it does not change the times to an appreciable extent. 


VERB-OBJECT TEST 


I II Easiest 
sing read wash 
build tear sing 
wear throw bake 
shoot paint read 
scold mail chew 
win light learn oer 
answer sail mail 
weave spin sweep 
wink lock scold 
mend wash wear 
pump bake sharpen 
learn spill kiss 
open kiss smoke 
eat polish answer 
climb sweep climb 
lend fill lock 
smoke sharpen | throw 
singe write sail 
dig chew dig 
sift drive wink 


3. The supraordinate concept or species-genus test. The in- 
structions are to name a class to which the given object belongs, 
or to “tell what sort of thing each is”; as ““oak—tree.” 

4. The subordinate concept or genus-species test. The in- 
structions are to name an example of the class mentioned, or 
to “mention a—”, as, “color—red”’. 

5. The part-whole test. The instructions are to name the 
whole thing of which the part is mentioned ; as “elbow—arm”. 
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Supraordinate 
Concept Test 
oak 
measles 
July 
shark 
quinine 
beef 
canoe 
banana 
Atlantic 
Alps 
penny 
dictionary 
cabbage 
Rhine 
murder 
dog 
sparrow 
London 
football 
rose 


Whole-Part 
Test 
apple 
clock 
knife 
book 
hat 
pencil 
hand 
dog 
oyster 
church 
chair 
bird 
banana 
shoe 
train 
finger 
house 
coat 


cart 
face 
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Subordinate 
Concept Test 

color 

holiday 

fish 

tool 

metal 

vegetable 

coin 

city 

insect 

food 

fruit 

disease 

grain 

drink 

month 

ocean 

language 

river 

newspaper 

tree 
Agent-Action Action-Agent 

Test Test 

baby gallops 
fire bites 
dog boils 
laborer sleeps 
pencil floats 
army growls 
heart sails 
pin roars 
gun scratches 
eyes stings 
bird shoots 
wind melts 
lungs swims 
bell explodes 
musician aches 
parrot blows 
clock mews 
axe cuts 
broom flies 
mosquito burns 


Part-Whole 
Test 


elbow 
hinge 
page 
finger 
wing 
morning 
blade 
mattress 
chimney 
cent 
sleeve 
brick 
deck 
France 
pint 

fin 
steeple 
month 
hub 
chin 


Attribute- 
Substance 
Test 


sharp 
hot 
dusty 
raw 
deep 
ripe 
funny 
tall 
stormy 
new 
hilly 
strong 
muddy 
pretty 
noisy 
white 
steep 
round 


smoky 


curly 
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6. The whole-part test. The instructions are to name a part 
of each thing mentioned ; as “apple—core”’. 

7. The agent-action or subject-verb test. The instructions are 
to put an appropriate verb to each noun as subject; or to “tell 
what each of these does or can do;” as “baby—cries”. 

8. The action-agent or verb-subject test: The instructions are 
to supply a subject to each verb, or to “tell what does or can do 
each of these things;’ as, “horse gallops.” 

9. The attribute-substance or adjective-noun test. The in- 
structions are to supply an appropriate noun for each adjective, 
or to “tell something that is or may be each of the following”, 
or to complete the expression, “A good—”,, etc. ; as, “sharp knife’, 

10. The mixed relations test. In the preceding tests, the task 
remains the same through a series of reactions; in the present 
test the particular relation along which the reaction is required 
to occur changes with each reaction—the object being to get 
some insight into flexibility of mental performance. We were 
long at a loss for some means of indicating the new task without 
lengthy explanations at each new stimulus and also without 
the use of such technical terms as supraordinate, etc. Finally a 
device used by one of us previously in the study of consciousness 
of relations seemed to meet our present needs: the relation along 


which the reaction is to take place is indicated before each new 


stimulus word by a pair of words serving as a sample. The sub- 
ject is to note the relation of the second word to the first, and 
then find a word standing in this same relation to the third word. 
Thus, in the example “Box—square Orange—?” “square” gives 
a quality of “box’’, or, more specifically, the shape of the box, 
and it is required to mention the shape of an orange; in the 
example ““East—west Day—?” since east and west are opposites 
the task is to find the opposite of day ; and in the example, “Penny 
—copper Nail—?” the task is to mention the material of which 
the nail is composed. Some of the relations are not readily nam- 
ed, but little difficulty has appeared, with the adult educated sub- 
jects already tested, in grasping the relation from the sample 
given. Instructions for this test must proceed largely by 
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the use of samples, of which several must be given, in order that 
the subject may realize that it is not always the same relation that 


is needed, but a new relation each time as indicated by the first 
two words in the line. 


Mixed Relations Test Mixed Relations Test 
I 7 II 

Eye—see Ear— Good—bad Long— 
Monday—Tuesday April— Eagle—bird Shark— 
Do—did See— Eat—bread Drink— 
Bird—sings Dog— Fruit—orange Vegetable— 
Hour—minute Minute— Sit—chair Sleep— 
Straw—hat Leather— Double—two Triple— 
Cloud—rain Sun— England—London France— 
Hammer—tool Dictionary— Chew—teeth Smell— 
Uncle—aunt Brother— Pen—write Knife— 
Dog—puppy Cat— Water—wet Fire— 
Little—less Much— He—him She— 
Wash—face Sweep— Boat—water Train— 
House—room Book— Crawl—snake Swim— 
Sky—blue Grass— | Horse—colt Cow— 
Swim—water Fly— : Nose—face Toe— 
Once—one Twice— Bad—worse Good— 
Cat—fur Bird— | Hungry—food Thirsty— 
Pan—tin Table— Hat—head Glove— 
Buy—sell Come— Ship—captain Army— 
Oyster—shell Banana— Man—woman Boy— 


Results with the logical relations test. After the tests had 
reached practically their present condition, they were tried with 
thirteen college and graduate students (in a few cases, the num- 
ber of individuals was less than this). In these experiments, 
lists of ten stimulus were presented visually, but the time of the 
single reactions was roughly taken by the device mentioned on 
page 17. Usually two, and in the case of the opposites and verb- 
object tests four lists of ten were used, and each subject’s aver- 
age time per single reaction was obtained. The averages given 
in the accompanying table are the average of the individual aver- 
ages, and the A.D. is that of the individual averages from the 
general average. 
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Av. per A. D. of indivs. Range 

single reaction P.E. from generalav. of indivs. 
Opposites 1 and 2....... 1.23 : .16 1.03—1.50 
Opposites, easiest ...... I.1I 04 12 0.85—1.40 
Verb-obj. 1 and 2....... 1.39 05 .19 1.08—1.75 
Verb-obj., easiest ...... 1.31 05 14 1.10—1.55 
‘Suproord. concept....... 1.54 .07 31 0.90—2.20 
Subord. concept......... 1.84 .07 31 1.20—2.63 
we 1.53 .06 .27 1.03—2.50 
1.57 .07 .32 1.13—2.35 
Agent-action ..........- 1.30 .03 12 0.93—1.70 
Action-agent ........... 1.55 .07 .32 1.03—2.68 
Attrib.-subst. ........... 1.53 .07 .28 1.08—3.05 
Mixed relations ........ « Qs .13 53 2.33—4.40 


The degree of agreement between the results of the several 
logical relations tests is a matter of some interest as indicating 
to what extent a single test is a fair indication of the individual’s 
ability in this whole class of performances. By methods which 
will be more fully described in another paper, we have determined 
the average standing of each of our thirteen subjects in the nine 
logical relations tests (excluding the mixed relations test), and 
have correlated this average standing with the standing in each 
single test. The results follow, in the form of Pearson coefh- 
cients, uncorrected for attenuation. 


r P.E. 
Correlation of Average with: Opposites................ +.88 .03 
+.70 08 

Subordinate conc. ...... +.72 

Supraordinate concept.... +.91 .03 

. +.86 04 

Action-agent ............ +.84 04 

Attribute-substance ...... +.54 12 


As far as these few results indicate, then, the opposites and 
supraordinate concept tests seem slightly better than the rest as 
representative of this general sort of controlled association. 
The correlation between the opposites and the supraordinate 
concept tests was +.70, with P.E. of .08, while the average 
correlation between any two of the nine logical relations tests 
is +.57. 
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Comparative speed of the different forms of controlled 
association 


It may be of interest, since results are available in several tests 
from comparable and highly reliable (though not numerous) sub- 
jects, to bring together the times per single reaction, placing them 
in the order from quickest to slowest. In all these cases, a series 


of stimuli was simultaneously presented, so that overlapping took 
place. 


Time in seconds 


Performance. _ per single reaction. P.E. 
Reading letters or short words 


Adding two one-place numbers (Kraepelin blank )..... 1.07 08 
Adding 4 (const. increment test)..............0..0eeeee 1.36 .08 
Subtracting 4 (const. increment test) 1.53 .07 
Adding 17 (const. increment test) .................06- 3.90 31 


These differences are of course not to be understood as mean- 
ing that the finding of opposites is always, or even on the aver- 
age, quicker than the finding of wholes when parts are given. 
The times for opposites that are by no means recondite or unusual 
run up to an average of at least 5 seconds per reaction. It would 
be futile to attempt to determine the average or median time for 
all opposites, and even more futile to make such an attempt in 
case of the part-whole, genus-species and many other relations; 
there would be no way of setting an upper limit to the difficulty 
of the single stimulus words. Such a statement as that the mind 
passes more readily from species to genus than from genus to 
species has therefore not much real validity. The fact simply is, 
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as far as our results are concerned, that the easiest opposites are 
easier than the easiest part-whole associations, etc.; and by “eas- 
iest” is meant, in case of the several logical relations tests, the 
twenty easiest. The differences between the speed of controlled 
associations are perhaps mainly dependent on the factor of fre- 
quency in past experience, and especially on the frequency of 


linguistic transitions. Thus, transitions between opposites are fre- 


quent in common speech, and many pairs of opposites thus be- 
come verbally associated in a high degree. The reproductive 
tendencies in case of the most commonplace opposites are there- 


fore strong; and it may also be that the “mental set”, or “de- | 


termining tendency”, is better drilled in case of finding opposites 
than in many other sorts of logical relation. 
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VIII. THE UNDERSTANDING OF INSTRUCTIONS 


As already mentioned (p. 20), a test should not ordinarily be 
begun till the subject certainly understands the instructions ; other- 
wise the time measured is partly occupied with grasping the 
problem, and only partly with its execution. Each test should, as 
nearly as possible, be a test of one sort of performance. But it 
seemed desirable to attempt to test the ability to understand 
instructions, and accordingly efforts were made to prepare a test 
which should give many different sets of very simple instructions, 
with the object of discovering the subject’s speed in apprehend- 
ing them. After much experimenting, the following were pro- 
duced, and the test was named the directions test.. This test 
should, we believe, be given as a list or continuous test, with 
rough timing also of the single reactions, so as to get the median 
as well as the average time of response. The reactions are to 
be made with a pencil; and the test can very well be made with 
a time limit as well as with an amount limit. 

The conditions which it was sought to meet in the test ma- 
terial are (1) that the motor response should be very simple 
and quickly performed; (2) that the instructions should be very 
simple, but varied; and (3) that the instructions should be as 
concise as possible, in order that reading time might not be the 
determining factor. 

1. Easy directions test. Two blanks are provided, of ap- 
proximately equal difficulty, according to the results so far in 
hand. The halves are also approximately equivalent. 

2. Hard directions test. The object here is to complicate the 
directions somewhat, by calling for conditional and alternative 
responses, etc. The blank is arranged in the general form of an 
Ebbinghaus combination test. The instructions are simply to fill 
in the blank according to the directions in it. The first two or 
three directions are easy, so as to put the subject on the right 
track. The remaining units within the blank (except the last) 
are so chosen as not to be very unequal, with the object of making 
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the blank available for use with a time limit. It can not, how- 
ever, be claimed for this test that it is as well worked over and 
standardized as the others in this series. 

Results with the directions tests. Data so far in hand are 
rather meager, eight subjects having taken the easy directions 
test in approximately its present form, and six subjects the 
harder test—all educated adults. The results follow: 


DIRECTIONS TESTS. 
AY PE. A.D. Range 
Easy tests, time in secs. per reaction.. 3.60 28 92 2.30—5.70 
Hard test, time for whole blank .....107.6 6.0 18.4 76—134 


If the number of reactions in the hard test is counted as 20 
(which is approximately correct), the average time per re- 
actions is 5.38 seconds; the reactions are no doubt slower 
in this than in any other of the tests described in this paper. To 
judge from the six subjects who have taken both the easy and 
the hard directions test, the correlation between the two is very 
high (Pearson I = -+ .92). 
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Cross out the smallest dot: @ 


Put a comma between these two letters: G H 
How many ears has a cat? 
Make a line across this line: 


Show by a cross which costs more: a hat or an orange. 


Write 8 at the thinnest part of this line: === 
Write any word of three letters. 


Put a dot in one of the white squares : 


Cross out the word you know best: fish, brol, matzig. 


Leave this just as it is: >>>——> 


How many t’s are there in twist? 

Dot the line that has no dot over it: | | | 
Write o after the largest number: 3 86 12 
Mark the name of a large city: London, painter.’ 
Make a letter Z out of this: vd 

Join these two lines: 


Write s in the middle square : 


Write any number smaller than to. 


Put a question mark after this sentence 
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Cross out the g in tiger. 

Write 2 between the two dots: «6 © <== 
How many feet make a yard? 

Write + over the longest word: It rained yesterday. 


Put a dot below this line: 


Write the sum of these numbers: 3 


Make a boy’s name by adding one letter to Joh 


Make a cross in the circle: TAS G) a 


What comes next after D in the alphabet ? 


Write 7 in the largest square: 


Cross out the blackest letter in TEXAS 


Write g on the egg-shaped figure: G) © ese 
Make two dots between these lines: an 


Put the sign = where it belongs: 3+2 5. 
Write here.....the middle letter of get. 


Put a nose on this face: t. ) 
XXX 


Add a cross and make these rows equal: x x x X 
Put a dot in the circle, below the center : (:) 


Draw a line around the three dots: « « pe 


Cross out the last word in this sentence. 
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With your pencil make a dot over any one of these 
letters F G H I J, and a comma after the 
longest of these three words: boy mother _ girl 


Then, if Christmas comes in March, make a cross right 


hete:....... but if not, pass along to the next question, and 
tell where the sun rises................0.00.. If you believe that 
Edison discovered America, cross out what you just 
wrote, but if it was some one else, put in a number to 
complete this sentence: ‘A horse has............ feet.” 
Write yes, no matter whether China is in Africa or not 
ik eta ; and then give a wrong answer to this question: 
“How many days are there in the week?”’...................... 


Write any letter except g just after this comma, and 


Now, if Tues- 


day comes after Monday, make two crosses here 


then write zo if 2 times § are 10 


but if not, make a circle here............ or else a square here 


Be sure to make’ three crosses between these 


two names of boys: George.................. Henry. Notice 
these two numbers: 3, 5. If iron is heavier than 
water, write the larger number here............ , but if iron 


is lighter write the smaller number here 


Show 


by a cross when the nights are longer: in summer?...... 


in winter?........ Give the correct answer to this ques- 
tion: “Does water run uphill?” and repeat 
your answer here................ Do nothing here (5 + 7 = 


ileal ), unless you skipped the preceding question; 
but write the first letter of your first name and the last 


letter of your last name at the ends of this line: 
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IX. THE FREE ASSOCIATION EXPERIMENT 


Few procedures in experimental psychology have so richly re- 
warded their investigators with the possibilities of practical ap- 
plication. In ordinary psychological nomenclature, it is the “as- 
sociation” experiment par excellence. Within the past seven 
years it has achieved, and bids fair to hold indefinitely its place 
in the foremost rank among the methods of individual psycho- 
logy. The body of work that has gathered about it is probably 
greater than that about any other single psychological experi- 
ment, and it is not surprising that it constitutes one of the best 
understood, as well as one of the most potentially significant of 
them. 

The preliminary task of standardization is to provide as er- 
ror-free a method as practicable, but the main object of stand- 
ardization is to afford a basis for making comparisons between 
different individuals. An experimental method becomes stand- 
ardized in the most complete sense when, given a proper technique, 
it is possible to accurately rate individual records with refer- 
ence to an empirical scale. None of the “mental tests” possesses 
this quality to a degree comparable with the free association ex- 
periment, within the limits of the English language. This is main- 
ly due to the work of Kent and Rosanoff which established a defi- 
nite standard of normality for a specific association material.? 
Within the bounds of its application, it would be an impertinence 
to offer as “standard” any procedure for the free association 
test other than the one which these authors have developed; our 
first endeavor will be then to describe this experimental material, 
and to indicate what seem to be the best methods for its ap- 
plication. 

The Kent-Rosanoff experiment consists of one hundred or- 
dinary English words of somewhat varying difficulty, in the 
order given on the opposite page, and the making of the test 


*Kent and Rosanoff, A Study of Association in Insanity, Am. Journal of 
Insanity, LX VII, pp. 37-96 and 317-390. 
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requires, according to the subject, from ten to twenty-five 
minutes. 

This experiment was made by Kent and Rosanoff with 1000 
normal subjects, and the responses were tabulated to each in- 
dividual stimulus word. These constitute the so-called “Fre- 
quency Tables,” and their use is to determine the “value,” in 
terms of its frequency coefficient, for any reaction or series of 
reactions in a record of this experiment. After each response- 
word in the tables occurs a number, which is the number of times 
that the word to which it attaches occurred as a response to the 
stimulus word among the 1000 observations. This figure, divided 
by 10, is taken as the “value” of the response. Thus the “value” 
of the association table-accommodation is .1, because it was given 
by only one of the 1000 subjects; that of table-chair is 26.7, being 
given by 267 of the 1000 subjects, that of dark-room is 2.2, that 
of music-art is .7. It is found, then, that different records of the 
test show marked differences in the “value” or usualness of the 
associative responses. By means of these frequency tables, the 
proper “value” is assigned to all reactions obtained from the 
stimulus-words. Any one wishing to work with this experiment 
must provide himself with a copy of the tables, which it is 1m- 
possible to reproduce here. Some measure of central tendency 
for all the measures should be taken, and the distribution of 
the measures indicates the median to be preferable for this pur- 
pose to the average, aside from its greater ease of calculation. 

The first and foremost datum of the Kent-Rosanoff experi- 
ment is an empirical measure of the tendency of the subject’s 
train of thought to move in usual or individual channels; more 
accurately speaking, along objective or subjective lines. A 
number of interpretational questions arise in connection with 
this finding, which seems less correlated with education than 
with temperament. It is perhaps the best objective correlate 
of temperament at present to hand, but the matter is a rather 
complicated one, more suitable for separate discussion. Here 
need be emphasized only the preciseness and objectivity with 


*American Journal of Insanity, LXVII, pp. 48-90. To be had of G, E. 
Stechert & Co., New York. 
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1. Table 


26. 


Wish 


. Bitter 


2. Dark 


27. 


River 


52. Lamp 


77. Hammer 


3. Music 


28. 


White 


53. Dream 


78. Thirsty 


4. Sickness 


29. 


Beautiful 


54. Yellow 


79. City 


5. Man 


30. 


Window 


55. Bread 


80. Sqnare 


6. Deep 


31. 


Rough 


56. Justice 


81. Butter 


7. Soft 


32. 


Citizen 


57. Boy 


82. Doctor 


8. Eating 


33. 


Foot 


58. Light 


83. Loud 


9. Mountain 


34. 


Spider 


59. Health 


84. Thief 


10. House 


35. 


Needle 


| 60. Bible 


85. Lion 


11. Black 


36. 


Red 


61. Memory 


86. Joy 


12. Mutton 


37. 


Sleep 


62. Sheep 


87. Bed 


13. Comfort 


38. 


Anger 


63. Bath 


88. Heavy 


14, Hand 


39. 


Carpet 


64. Cottage 


89. Tobacco 


15. Short 40. Girl 65. Swift 90. Baby 
16. Fruit 41. High 66. Blue 91. Moon 
17. Butterfly 42. Working 67. Hungry 92. Scissors 


18. Smooth 


43. 


Sour 


68. Priest 


93. Quiet 


. Command 


44, 


Earth 


69. Ocean 


94. Green 


. Chair 


45. 


Trouble 


70. Head 


95. Salt 


. Sweet 


46. 


Soldier 


71. Stove 


96. Street 


22. Whistle 


47. 


Cabbage 


72. Lone 


97. King 


23. Woman 


48. 


Hard 


73. Religion 


98. Cheese 


24. Cold 


49. 


Eagle 


74. Whiskey 


99. Blossom 


25. Slow 


50. Stomach 


75. Child 


100. Afraid 
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which it is possible to evaluate an experiment of such intimate 
and subjective character. 

If special circumstances render it desirable, it is possible to 
employ only a larger portion of the hundred words for deter- 
mination of the usualness in response, substituting for the re- 
mainder, words adapted to the special situation in hand. It 
would be desirable indeed, if the Kent-Rosanoff experiment were 
made the framework of all experiments for Tatbestandsdi- 
agnostik, the individually significant words being either added to 
it or replacing some, not over 10 or I5 per cent. of its con- 
stituents. To deal objectively with questions of Tatbestands- 
diagnostik requires a number of precautions in the construction 
of the special series, the enumeration of which would be out of 
place here, and which are fully discussed by the investigators 
of this application of the method. 

Unfortunately, determinations of the “median of community” 
(i. e., the median “value” of the 100 reactions in a record) have 
as yet been made in only a limited nnmber of subjects. In some 
pathological cases it would become indefinitely small; the lowest 
median ever observed by either of the writers in a normal sub- 
ject is .7. Such a figure would mean that half the reactions of 
this record were of a frequency below that of the reaction music- 
art quoted above. The other extreme of the range, so far as 
observed, is 18.2,-i. e., half of the reactions in such a record are 
more common than, i. e., music-piano. The general average 
value of the reactions in the above mentioned records lies not 
far from 9.0, that is, about the frequency of a reaction such as 
mountain-valley. 

The present experimental method is placed under one disad- 
vantage to a much greater degree than other association tests; 
its material cannot be repeated within an ordinarily practicable 
time save under greatly changed essential conditions. One can 
foresee that circumstances may arise in which a comparative 
study with material of greater extent is desirable. Provision is 
here made for such material to be available,’ but with a change in 
the character of the material comes inevitably a change in the 
method of evaluation. Beyond the range of the frequency tables 


* See. Appendix, pp. 80 ff. 
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one must fall back on the quasi-logical system of-classifying the 
associations that was practically the sole means of dealing with 
such material until the data on statistical frequency were com- 
piled. The proper function of the test, however, is the same as 
before, and so is the object of its evaluation: the measure of 
egocentricity in the responses. | - 

There is no need to fully repeat the remarks in a previous con- 
tribution regarding the method of evaluation that seems best 
adapted to these conditions. It is a five-fold classification, in- 
cluding categories termed (1) the egocentric, (2) the supraor- 
dinate, (3) the contrast, (4) the miscellaneous or “internal ob- 
jective,” and (5) the speech-habit.? 

‘For ordinary purposes of comparison, the principal question 
concerns the number of reactions that fall into the category of 
the egocentric; and a large or small number of such associations 
is subject to analogous interpretations with the empirically de- 


termined tendency towards common or individualized responses. 

*The definitions and illustrations of the categories may be summarized 
from the previous paper as follows: 

1. The egocentric reactions may be typified by— 

a. Predicate reactions. Cloud-ominous, flower-pretty, crooked-line, red- 
rose, scratch-cat, lion-roar, money-wish, invent-machine, weasel-stealth, beau- 
ty-rose, safe-quite, almost-grown, sing-well, never-decide, nicely-very (includ- 
ing the responses yes and no). 

b. Responses in the form of proper names. Citizen-New York, boy- 
Johnny, mountain-Kearsarge. 

c. Reactions interpreting the stimulus word as a proper name. LEagle- 
newspaper, park-square. 

d. Reaction involving the response of a pronoun. Hand-you, health-me. 

e. Interjections, failures of response or repetitions of the stimulus word. 

2. The supraordinate category is confined strictly to the individual-genus 
order, defined in such examples as, priest-man, potato-vegetable, lily-flower, 
cow-animal. 

3. The contrast group is composed, of course, of reactions in which the 
response meets the opposite of the stimulus and is made up of such associa- 
tions as, good-bad, trouble-pleasure. scatter-gather, fertile-sterile, and the 
like. 

4. The miscellaneous category is composed essentially of the remaining 
reactions of the “inner” type. It includes about 45% of all associations. 

5. The speech-habit group is composed of associations by familiar phrase 
(stand-pat), word compounding (play-ground), simple sound associations 
(tease-sneeze) and syntactic changes (high-height). (Psychol. Review, 1911, 
18, 229-288. ) 
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The egocentric is the most variable of the five categories, ranging 
from next to none to more than half of the total reactions in a 
single experimental series. 

With respect to timing the associations, the stopwatch is the 
almost universal method, and must be accepted as such, until 
some more accurate procedure is devised. Significant differences 
are usually coarse enough to be manifested in measures of no 
greater precision than this. More uniformity in the manipulation 
is desirable; at present, some operators start the watch on the 
accented syllable, others at the first syllable, of the simulus word. 
The watch should always be stopped at the first indication of 
response, even if it does cause occasional failure of timing through 
the subject’s clearing his throat. 

Individual differences in association time should be discussed 
from the standpoint of the distribution rather than any single 
measure. The median is rather preferred as a measure of central 
tendency, though for practical purposes, its advantages over the 
average are of less account here than in most cases of skew 
distribution. The presence of many and exclusively long meas- 
ures happens to be more important here than in most similar 
series of measures. Jung has proposed a special comparison of 
the average and median; this’is a convenient statement of the 
distribution, but it is not an index of emotivity, beyond the 
limited extent to which the association time can be interpreted 
in this direction. 

The usual instruction in the free association test is that the 
subject shall reply with the first word the stimulus suggests to 
him, but in ordinary practise this is not rigidly enforced, it being 
sometimes possible to derive elements of special significance from 
factors that determine the subject’s departure from the set in- 
structions. For comparison with the frequency tables, it is 
readily apparent that the single word response must be rigidly 
required in the Kent-Rosanoff experiment; in cases of derelic- 
tion from this rule, it is the practise of these authors to repeat 


* Coughing at such times has received some notice as a Komplexmerkmal, 


though it has been sagely remarked that this loses much of its significance 
if the subject in question has a cold. 
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the stimulus word at the end of the experiment, in order to 
obtain a reaction of the required character. 

Presentation and response have nearly always been oral in 
this experiment, and there is good reason to make no change in 
them. Experimenters differ as to the manner in which they 
modulate the voice to the test. Some experimenters, as Dr. 
Rosanoff, speak the stimulus word with a rising inflection, as 
though asking a direct question, some as an exclamation, as 
though endeavoring to hurl the subject’s “complexes” in his 
teeth, others ina monotone. There are subjects whose responses 
might be somewhat affected by these differences in procedure. 
This is one of the minor features of the test, in which, though 
uniformity among different investigators is scarcely practicable, 
the experimenter should at least strive to keep his own procedure 
constant. 

And while it is not proven, it is most probable that the re- 
sponses are also affected by the personality of the operator. In 
making fine comparisons between records by different experi- 
menters this fact must be borne continually in mind. 

So far as developed, the continuous form of the free associa- 
tion experiment is a method of possibilities rather than of 
promise. It would probably be capable of performing many of 
the functions of the discrete form, but there are external rea- 
sons why it would not be likely to perform them so well. The 
problem of standardization would be practically confined to the 
development of the most significant scheme of evaluation after 
the pattern indicated by previous workers with the test. It 
might be possible to employ the scheme of evaluation proposed 
for the discrete free association experiment, each single word 
given being allowed to serve as the stimulus word for the next 
association. So far as normal psychology is concerned the 
method has thus far dealt very largely with group averages. 
As a method of individual psychology it may assume a position 
comparable to its better known congener only as a result of 
extended and laborious researches. 


Note: Blanks for all the preceding tests are to be obtained from the 
C. H. Stoelting Company, 113-125 North Green St., Chicago. 
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The following series of 1000 words is intended for general use 
in the free association experiment. It is a revision of the series 
employed in the experiments on the practise effects in the test, 
and is modified in the manner suggested by the experience of this 
investigation. It is intended to contain 1000 different words, 
none over three syllables, so far as possible familiar and un- 
ambiguous. It is not far from exhausting the total available 
number of such stimulus words. Ambiguous stimulus words 
have a special and useful purpose, but not in a test of the present 
character. The details of the preparation of this list were sub- 
stantially the same as in the previous list, save in one particular. 
The division into twenty series of fifty words each is followed 
here. But the present list also contains the hundred words of 
the Kent-Rosanoff series, distributed pro rata, five words in each 
series, and in their actual order of sequence in the Kent-Rosanoff 
test; otherwise their arrangement in the series is random, save 
that none occur in the first ten words of a series. In the list as 
printed, the words from the Kent-Rosanoff series are distin- 
guished from the remainder by an asterisk, and the associations 
of these words may be evaluated by the frequency tables. In 
using single series of fifty words it is recommended that a sheet 
of paper of fifty lines be obtained upon which several records 
of reactions to the same stimuli may be conveniently noted. 
Stimulus words not evoking a reaction according to instructions 
may be repeated at the close of the series; and if a stimulus word 
evokes, as a response, the word coming next in the series, this 
word is omitted and given at the close. 

The complete list is as follows: 
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I 
bottle 
prodice 
rope 
delicate 
thick 
end 
omelet 
expensive 
cap 
barrel 
burglar 
design 
cry 
hip 
overcoat 
freeze 
*table 
lightning 
follow 
parlor 
smoke 
stretch 
tar 
snake 
purpose 
*dark 
unfair 
ditch 
tiger 
*music 
wicked 
prefer 
fish 
instrument 
guilty 
seed 
*sickness 
crush 
rich 
hash 
unseen 
death 
umbrella 
blood 
gift 
*man 
allow 
sailor 
prospect 

school 


WORD LIST 


II 
drink 
captain 
cedar 
mischief 
clean 
also 
path 
ride 
salute 
grocery 
bashful 
true 
perverse 
occasion 
nuisance 
*deep 
pinch 
satisfy 
tank 
hat 
nourish 
sister 
*soft 
ham 
ugly 
age 
glory 
tough 
acid 
*eating 
crowd 
discourse 
watchful 
indecent 
exchange 
costume 
style 
trap 
*mountain 
drift 
crime 
cover 
abuse (v) 
open 
*house 
enjoy 
untrue 
dismay 
unburden 

again 


III 
locust 
divide 
restore 
tempt 
fade 
cheap 
compel 
power 
baker 
athlete 
*black 
roof 
cradle 
certain 
travel 
impréss 
daughter 
gun 
book 
barber 
natural 
elephant 
ostrich 
curse 
*mutton 
haste 
lizard 
result 
nonsense 
index 
fool 
dense 
life 
wine 
*com fort 
fever 
infirm 
comb 
spice 
starch 
venture 
*hand 
pirate 
brandy 
dress 
pebble 
adventure 
lip 
*short 
pint 
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IV 
weary 
tooth 
practise 
supper 
fun 
pepper 
best 
heart 
island 
machine 
pit 
*fruit 
return 
marriage 
marsh 
owl 
water 
summer 
copper 
beetle 
statue 
clothes 
oblong 
*butterfly 
constable 
cloud 
collapse 
solid 
number 
goose 
railroad 
excite 
hornet 
*smooth 
delay 
begin 
cat 
asylum 
knee 
tight 
car 
*command 
insect 
hope 
insist 
*chair 
star 
ice 
picture 

bind 
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V 
room 
pencil 
dig 
indiscreet 
restless 
simple 
measure 
loss 
reckless 
flirt 
pause 
prosper 
*sweet 
avoid 
fresh 
real 
potato 
*whistle 
bite 
clear 
eternal 
jealous 
barn 
*woman 
persuade 
merit 
receive 
above 
conceal 
revolt 
*cold 
join 
tender 
offense 
guide 
prompt 
floor 
advance 
bundle 
ignorant 
*slow 
blunder 
confidence 
knob 
flesh 
future 
wart 
market 
chocolate 
disease 


VI 
crown 
get 
honest 
vacant 
beechnut 
splinter 
unbelief 
argue 
conflict 
alike 
skin 
inside 
*wish 
hero 
scarlet 
lamb 
neck 
clasp 
spear 
*river 
ox 
serious 
garter 
key 
conquest 
*white 
scratch 
cool 
correct 
paste 
uncertain 
pudding 
*beautiful 
bacon 
rancid 
fertile 
dog 
perfume 
toy 
dust 
pansy 
lake 
*window 
vow 
pancake 
cork 
gay 
feast 
gem 
enormous 
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VII 


forget 


goat 
pulse 
unwholesome 
attention 
dairy 
boast 
color 
chin 
servant 
dislike 
dead 
*rough 
fortune 
candy 
perfect 
disdain 
fierce 
constant 
violent 
care 
indeed 
*citizen 
death 
sparrow 
*foot 
over 
bother 
forward 
prepare 
establish 
gold 
along 
cannon 
boat 
song 
*spider 
another 
irony 
art 

dove 
poem 
herald 
*needle 
treasure 
sensitive 
fog 
tunnel 
remain 
closet 


VIII 
pot 
camp 
shirt 
chain 
adult 
violin 


reason 
excuse (n) 
roast 

fig 

face 

*red 
common 
complexion 
deserve 
dim 

view 
*sleep 
dirt 

shoe 
slave 
protect 
sting 
funny 
solemn 
little 
*anger 
ramble 
family 
annoy 
confusion 
ripe 
greasy 
admire 
cup 

insult 
easy 
impudent 
*carpet 
decay 
*girl 
announce 
prudent 
trumpet 
convenient 
ask 
supreme 
portable 
before 
alone 
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IX 
accuse 
flame 
clown 
edge 
frost 
corrupt 
appear 
caution 
poverty 
curtain 
army 
*high 
people 
polish 
*working 
almost 
idea 
cask 
alcohol 
minute 
stain 
nurse 
*sour 
chapel 
plant 
sky 


forest 


companion 


repeat 
against 
finish 
sermon 
*earth 
conceit 
crack 
drag 
condemn 
iron 
emperor 
plead 


*trouble 
mouse 
event 
claw 
ingenious 
minister 
impose 


WORD LIST 


riot 
preserve 
preach 
unclean 
steep 
master 
sulphur 
laugh 
imp 
invite 
raisin 
minnow 
promise 
ungracious 
good 
*soldier 
mask 
money 
doll 
rotten 
*cabbage 
cost 
irksome 
apricot 
stone 
*hard 
brute 
escape 
gain 
trifle 
*eagle 
admit 
snow 
about 
equal 
brown 
late 
adore 
perish 
tomb 
harsh 
*stomach 
wasp 
unripe 
friend 
taste 
joke 
propose 


XI 
uphold 
pickle 
food 
raven 
unwell 
ready 
away 
blame 
competent 
mark 
improve 
*stem 
raw 
defy 
brook 
vile 
storm 
refined 
thankful 
fast 
tremble 
center 
*lamp 
saddle 
pin 
denounce 
cook 
fraud 
bring 
paint 
nut 
immense 
*dream 
condition 
descend 
splash 
abroad 
*yellow 
deceive 
bride 
worship 
infamous 
drop 
*bread 


backwards 


pattern 
cliff 
level 
body 
elevate 


XII 
intimate 
able 
suspect 
barley 
attack 
dishonor 
accident 
betray 
door 
prince 
*justice 
aim 
revenge 
active 
purple 
decoy 
noise 
fable 
unsafe 
fame 
strength 
scoff 
humble 
*boy 
interest 
old 
wealthy 
modest 
*light 
fact 
violet 
appetite 
attraction 
*health 
across 
piano 
least 
salmon 
price 
garden 
scar 
burn | 
ashamed 
*bible 
deny 
quantity 
idle 
wash 
reproach 


energy 
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XIII 
alarm 
distrust 
dimple 
bounce 
twig 
indulge 
run 
agree 
secret 
ache 
advantage 
napkin 
hill 
shelter 
name 
injure 
*memory 
finger 
emblem 
spool 
unfit 
middle 
*sheep 
outrage 
accept 
low 
ardent 
*bath 
emerald 
wagon 
stun 
gentle 
dodge 
*cottage 
shock 
contrary 
hunt 
sin 
asleep 
exquisite 
sweat 
change 
*swift 
éxpert 
quality 
instant 
progress 
melt 
applause 
cream 
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XIV 
astonish 
vest 
whale 
outfit 
recover 
embrace 
devil 
game 
towel 
faithful 
dwarf 
use (v) 
north 
thread 
rejoice 
*blue 
disaster 
keep 
handsome 
rescue 
audacious 
cage 
honey 
guess 
disgrace 
shark 
flannel 
busy 
unmarried 
angel 

hospital 
secure 
*hungry 
jewel 
nice 
carve 
provoke 
alive 
*priest 
orange 
battle 
tube 
*ocean 
apology 
inch 
pretty 
brick 
verse 
*head 
bad 


XV 
contented 
praise 
pump 
poem 
tennis 
guard 
cake 


calm 


remorse 
play 
mouth 
linen 
*stove 
belt 
amuse 
sign 
bag 
concert 
*long 
absent 
maiden 
twist 
false 
plunge 
murder 
*religion 
magic 
believe 
author 
oil 
choke 
*whiskey 
silver 
noble 
breast 
person 
influence 
magnet 
glad 
ink 
introduce 
profane 
winter 
help 
repress 
*child 
field 
egg 
rat 
mock 


XVI 
find 
broom 
dagger 
love 
try 
lazy 
arrow 
come 
success 
coy 
pure 
shower 
jump 
*bitter 
uproar 
catch 
contest 
empty 
unhappy 
divine 
*hammer 
feather 
disorder 
naughty 
exacting 
abandon 
*thirsty 
incréase 
*city 
chase 

unemployed 
rhyme 
map 
wretched 
distance 
playful 
impulse 
land 
moderate 
velvet 
mix 
parent 
*square 
séparate 
sonnet 
trade 
nest 
fancy 
bench 
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XVII 
grief 
parsnip 
purse 
unlikely 
walk 
hod 
comrade 
thought 
lemon 
refuse 
paper 
cause 
pocket 
task 
kit 
*butter 
pie 
strong 
*doctor 
pig 
punish 
regiment 
walnut 
weather 
*loud 
remove 
fling 
compare 
queen 
same 
war 
play 
exercise 
grind 
*thief 
pretend 
kneck 
orchard 
president 
decent 
croak 
plaster 
lump 
question 
lend 
around 


merry 
*lion 

awake 
sacred 
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XVIII 
decorate 
chance 
sack 
scold 
portly 
sorrow 
month 
painful 
quarrel 
flower 
suffer 
fault 
*joy 
cab 
discord 
sponge 
mother 
*bed 
den 
support 
conscience 
devotion 
difficult 
adorn 
immoral 
spite 
brave 
circle 
lettuce 
ivory 
urge 
imagine 
infinite 
observe 
assist 
beast 
wheat 
*heavy 
repose 
terror 
under 
caress 
lard 
learn 
*tobacco 
destiny 
fire 
consent 
*baby 
excellent 


XIX 
contrast 
unhurt 
fix 
interval 
fond 
grain 
mistake 
front 
quart 
lecture 
*moon 
usher 
brier 
fountain 
church 
attempt 
feed 
tame 
medicine 
glove 
dispute 
*scissors 
evil 
irritate 
advice 
neighbor 
cravat 
*quiet 
entire 
contempt 
talk 
touch 
flag 
anxious 
hurt 
*green 
crumb 
éscort 
notch 
bird 
husband 
control 
stable 
*salt 
ornament 
errand 
hoop 
blush 
dull 
many 


XX 
demon 
credit 
frolic 
include 
rascal 
pardon 
soap 
fear 
arise 
cane 
destroy 
chart 
refresh 
*street 
wrong 
rattle 
*king 
send 
glow 
raft 
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